Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

TL;DR

This paper introduces a probabilistic Gaussian model for multi-modal test-time adaptation that explicitly models category-conditional distributions and uses adaptive contrastive rectification to improve prediction accuracy under distribution shifts.

Contribution

The paper proposes a novel probabilistic Gaussian model and adaptive contrastive asymmetry rectification for multi-modal TTA, addressing modality asymmetry issues.

Findings

Achieves state-of-the-art performance on diverse benchmarks.

Effectively models category-conditional distributions in multi-modal TTA.

Improves prediction calibration and decision boundary reliability.

Abstract

Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled target data during inference. Despite the documented success, the advancement of multi-modal TTA methodologies has been impeded by a persistent limitation, i.e., the lack of explicit modeling of category-conditional distributions, which is crucial for yielding accurate predictions and reliable decision boundaries. Canonical Gaussian discriminant analysis (GDA) provides a vanilla modeling of category-conditional distributions and achieves moderate advancement in uni-modal contexts. However, in multi-modal TTA scenario, the inherent modality distribution asymmetry undermines the effectiveness of modeling the category-conditional distribution via the canonical GDA. To this end, we introduce a tailored probabilistic Gaussian model for multi-modal TTA to explicitly model the category-conditional distributions, and further propose an adaptive contrastive asymmetry rectification technique to counteract the adverse effects arising from modality asymmetry, thereby deriving calibrated predictions and reliable decision boundaries. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts. The code is available at https://github.com/XuJinglinn/AdaPGC.