Surformer v1: Transformer-Based Surface Classification Using Tactile and Vision Features

TL;DR

Surformer v1 is a transformer-based model that effectively combines tactile and visual features for surface classification, achieving high accuracy and fast inference suitable for real-time robotic perception.

Contribution

This work introduces Surformer v1, a novel transformer architecture that fuses tactile and vision data for surface recognition, demonstrating superior speed and accuracy over existing models.

Findings

Surformer v1 achieved 99.4% accuracy in surface classification.

The model demonstrated an inference time of 0.77 ms, suitable for real-time applications.

Multimodal fusion improved classification performance compared to single modality models.

Abstract

Surface material recognition is a key component in robotic perception and physical interaction, particularly when leveraging both tactile and visual sensory inputs. In this work, we propose Surformer v1, a transformer-based architecture designed for surface classification using structured tactile features and PCA-reduced visual embeddings extracted via ResNet-50. The model integrates modality-specific encoders with cross-modal attention layers, enabling rich interactions between vision and touch. Currently, state-of-the-art deep learning models for vision tasks have achieved remarkable performance. With this in mind, our first set of experiments focused exclusively on tactile-only surface classification. Using feature engineering, we trained and evaluated multiple machine learning models, assessing their accuracy and inference time. We then implemented an encoder-only Transformer model tailored for tactile features. This model not only achieved the highest accuracy but also demonstrated significantly faster inference time compared to other evaluated models, highlighting its potential for real-time applications. To extend this investigation, we introduced a multimodal fusion setup by combining vision and tactile inputs. We trained both Surformer v1 (using structured features) and Multimodal CNN (using raw images) to examine the impact of feature-based versus image-based multimodal learning on classification accuracy and computational efficiency. The results showed that Surformer v1 achieved 99.4% accuracy with an inference time of 0.77 ms, while the Multimodal CNN achieved slightly higher accuracy but required significantly more inference time. These findings suggest Surformer v1 offers a compelling balance between accuracy, efficiency, and computational cost for surface material recognition.