Test-Time Adaptation for Tactile-Vision-Language Models

TL;DR

This paper introduces a reliability-aware test-time adaptation framework for tactile-vision-language models, improving robustness against modality shifts by estimating and leveraging modality-specific reliability signals.

Contribution

It proposes a novel reliability estimation method for each modality and integrates it into test-time adaptation, enhancing robustness under distribution shifts.

Findings

Achieves up to 49.9% accuracy improvement under severe modality corruptions.

Outperforms existing TTA methods on the TAG-C benchmark.

Demonstrates the importance of modality-wise reliability modeling for robustness.

Abstract

Tactile-vision-language (TVL) models are increasingly deployed in real-world robotic and multimodal perception tasks, where test-time distribution shifts are unavoidable. Existing test-time adaptation (TTA) methods provide filtering in unimodal settings but lack explicit treatment of modality-wise reliability under asynchronous cross-modal shifts, leaving them brittle when some modalities become unreliable. We study TTA for TVL models under such shifts and propose a reliability-aware framework that estimates per-modality reliability from prediction uncertainty and perturbation-based responses. This shared reliability signal is used to (i) filter unreliable test samples, (ii) adaptively fuse tactile, visual, and language features, and (iii) regularize test-time optimization with a reliability-guided objective. On the TAG-C benchmark and additional TVL scenarios, our approach consistently outperforms strong TTA baselines, achieving accuracy gains of up to 49.9\% under severe modality corruptions, underscoring the importance of explicit modality-wise reliability modeling for robust test-time adaptation.