OmniVaT: Single Domain Generalization for Multimodal Visual-Tactile Learning

TL;DR

OmniVaT introduces a novel framework for single domain generalization in multimodal visual-tactile learning, effectively bridging modality gaps and adapting to unseen domain shifts without multi-domain training.

Contribution

It proposes the first solution to SDG-VTL, integrating MFFA for modality alignment and DTG for domain adaptability, advancing multimodal VTL robustness.

Findings

Outperforms existing methods in cross-domain generalization

Effectively mitigates modality discrepancies without multi-domain data

Enhances adaptability to unseen domain shifts

Abstract

Visual-tactile learning (VTL) enables embodied agents to perceive the physical world by integrating visual (VIS) and tactile (TAC) sensors. However, VTL still suffers from modality discrepancies between VIS and TAC images, as well as domain gaps caused by non-standardized tactile sensors and inconsistent data collection procedures. We formulate these challenges as a new task, termed single domain generalization for multimodal VTL (SDG-VTL). In this paper, we propose an OmniVaT framework that, for the first time, successfully addresses this task. On the one hand, OmniVaT integrates a multimodal fractional Fourier adapter (MFFA) to map VIS and TAC embeddings into a unified embedding-frequency space, thereby effectively mitigating the modality gap without multi-domain training data or careful cross-modal fusion strategies. On the other hand, it also incorporates a discrete tree generation (DTG) module that obtains diverse and reliable multimodal fractional representations through a hierarchical tree structure, thereby enhancing its adaptivity to fluctuating domain shifts in unseen domains. Extensive experiments demonstrate the superior cross-domain generalization performance of OmniVaT on the SDG-VTL task.