Analytic Continual Test-Time Adaptation for Multi-Modality Corruption

TL;DR

This paper introduces MDAA, a novel method for multi-modal continual test-time adaptation that effectively handles domain shifts and multi-modal corruptions by mitigating catastrophic forgetting and dynamically fusing modalities.

Contribution

The paper proposes MDAA, combining analytic learning with dynamic late fusion, to improve multi-modal test-time adaptation under domain shifts and corruptions.

Findings

MDAA achieves state-of-the-art results on multi-modal TTA tasks.

The analytic classifiers effectively reduce catastrophic forgetting.

Dynamic late fusion improves reliability in multi-modal integration.

Abstract

Test-Time Adaptation (TTA) enables pre-trained models to bridge the gap between source and target datasets using unlabeled test data, addressing domain shifts caused by corruptions like weather changes, noise, or sensor malfunctions in test time. Multi-Modal Continual Test-Time Adaptation (MM-CTTA), as an extension of standard TTA, further allows models to handle multi-modal inputs and adapt to continuously evolving target domains. However, MM-CTTA faces critical challenges such as catastrophic forgetting and reliability bias, which are rarely addressed effectively under multi-modal corruption scenarios. In this paper, we propose a novel approach, Multi-modality Dynamic Analytic Adapter (MDAA), to tackle MM-CTTA tasks. MDAA introduces analytic learning,a closed-form training technique,through Analytic Classifiers (ACs) to mitigate catastrophic forgetting. Furthermore, we design the Dynamic Late Fusion Mechanism (DLFM) to dynamically select and integrate reliable information from different modalities. Extensive experiments show that MDAA achieves state-of-the-art performance across the proposed tasks.