Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities

TL;DR

This paper introduces TLV-CoRe, a novel collaborative learning framework that enhances the integration and generalization of tactile, language, and vision data in robotics, addressing sensor variability and improving cross-modal alignment.

Contribution

The paper presents a new CLIP-based method with a Sensor-Aware Modulator, tactile-irrelevant decoupling, and a Unified Bridging Adapter for improved multimodal tactile representation.

Findings

TLV-CoRe improves sensor-agnostic tactile feature learning.

The method enhances cross-modal alignment across modalities.

Experimental results validate the effectiveness of the proposed approach.

Abstract

Tactile sensing offers rich and complementary information to vision and language, enabling robots to perceive fine-grained object properties. However, existing tactile sensors lack standardization, leading to redundant features that hinder cross-sensor generalization. Moreover, existing methods fail to fully integrate the intermediate communication among tactile, language, and vision modalities. To address this, we propose TLV-CoRe, a CLIP-based Tactile-Language-Vision Collaborative Representation learning method. TLV-CoRe introduces a Sensor-Aware Modulator to unify tactile features across different sensors and employs tactile-irrelevant decoupled learning to disentangle irrelevant tactile features. Additionally, a Unified Bridging Adapter is introduced to enhance tri-modal interaction within the shared representation space. To fairly evaluate the effectiveness of tactile models, we further propose the RSS evaluation framework, focusing on Robustness, Synergy, and Stability across different methods. Experimental results demonstrate that TLV-CoRe significantly improves sensor-agnostic representation learning and cross-modal alignment, offering a new direction for multimodal tactile representation.