Bridging Modalities via Progressive Re-alignment for Multimodal Test-Time Adaptation

TL;DR

This paper introduces BriMPR, a novel framework for multimodal test-time adaptation that progressively re-aligns features across modalities to handle distribution shifts, improving model robustness in complex multimodal scenarios.

Contribution

The paper proposes a divide-and-conquer approach with prompt tuning and contrastive learning to effectively address multimodal distribution shifts during test-time adaptation.

Findings

Outperforms existing TTA methods on multiple benchmarks

Effectively aligns unimodal and cross-modal features

Enhances robustness to real-world domain shifts

Abstract

Test-time adaptation (TTA) enables online model adaptation using only unlabeled test data, aiming to bridge the gap between source and target distributions. However, in multimodal scenarios, varying degrees of distribution shift across different modalities give rise to a complex coupling effect of unimodal shallow feature shift and cross-modal high-level semantic misalignment, posing a major obstacle to extending existing TTA methods to the multimodal field. To address this challenge, we propose a novel multimodal test-time adaptation (MMTTA) framework, termed as Bridging Modalities via Progressive Re-alignment (BriMPR). BriMPR, consisting of two progressively enhanced modules, tackles the coupling effect with a divide-and-conquer strategy. Specifically, we first decompose MMTTA into multiple unimodal feature alignment sub-problems. By leveraging the strong function approximation ability of prompt tuning, we calibrate the unimodal global feature distributions to their respective source distributions, so as to achieve the initial semantic re-alignment across modalities. Subsequently, we assign the credible pseudo-labels to combinations of masked and complete modalities, and introduce inter-modal instance-wise contrastive learning to further enhance the information interaction among modalities and refine the alignment. Extensive experiments on MMTTA tasks, including both corruption-based and real-world domain shift benchmarks, demonstrate the superiority of our method. Our source code is available at https://github.com/Luchicken/BriMPR.