A Case Study on Visual-Audio-Tactile Cross-Modal Retrieval

TL;DR

This paper introduces VAT-CMR, a novel multi-modal retrieval model integrating visual, audio, and tactile data to improve robotic object retrieval, demonstrating superior performance through a case study.

Contribution

The paper presents a new three-modality cross-modal retrieval model with a dominant modality selection mechanism, enhancing retrieval accuracy in robotic applications.

Findings

VAT-CMR outperforms existing methods in retrieval accuracy.

Dominant modality selection significantly improves cross-retrieval performance.

Multi-modal fusion provides a more holistic object representation.

Abstract

Cross-Modal Retrieval (CMR), which retrieves relevant items from one modality (e.g., audio) given a query in another modality (e.g., visual), has undergone significant advancements in recent years. This capability is crucial for robots to integrate and interpret information across diverse sensory inputs. However, the retrieval space in existing robotic CMR approaches often consists of only one modality, which limits the robot's performance. In this paper, we propose a novel CMR model that incorporates three different modalities, i.e., visual, audio and tactile, for enhanced multi-modal object retrieval, named as VAT-CMR. In this model, multi-modal representations are first fused to provide a holistic view of object features. To mitigate the semantic gaps between representations of different modalities, a dominant modality is then selected during the classification training phase to improve the distinctiveness of the representations, so as to improve the retrieval performance. To evaluate our proposed approach, we conducted a case study and the results demonstrate that our VAT-CMR model surpasses competing approaches. Further, our proposed dominant modality selection significantly enhances cross-retrieval accuracy.