Audio-Visual Continual Test-Time Adaptation without Forgetting

TL;DR

This paper introduces AV-CTTA, a method for continual test-time adaptation of audio-visual models that enhances performance across changing domains without forgetting previous knowledge, by selectively updating fusion layer parameters.

Contribution

The paper demonstrates that adapting only the fusion layer improves domain transferability and proposes AV-CTTA, which dynamically retrieves and updates fusion parameters without source data access.

Findings

AV-CTTA outperforms existing methods on benchmark datasets.

Adapting only the fusion layer enhances cross-domain transfer.

The approach minimizes catastrophic forgetting during continual adaptation.

Abstract

Audio-visual continual test-time adaptation involves continually adapting a source audio-visual model at test-time, to unlabeled non-stationary domains, where either or both modalities can be distributionally shifted, which hampers online cross-modal learning and eventually leads to poor accuracy. While previous works have tackled this problem, we find that SOTA methods suffer from catastrophic forgetting, where the model's performance drops well below the source model due to continual parameter updates at test-time. In this work, we first show that adapting only the modality fusion layer to a target domain not only improves performance on that domain but can also enhance performance on subsequent domains. Based on this strong cross-task transferability of the fusion layer's parameters, we propose a method, $\texttt{AV-CTTA}$, that improves test-time performance of the models without access to any source data. Our approach works by using a selective parameter retrieval mechanism that dynamically retrieves the best fusion layer parameters from a buffer using only a small batch of test data. These parameters are then integrated into the model, adapted to the current test distribution, and saved back for future use. Extensive experiments on benchmark datasets involving unimodal and bimodal corruptions show our proposed $\texttt{AV-CTTA}$ significantly outperforms existing methods while minimizing catastrophic forgetting.