Shared & Domain Self-Adaptive Experts with Frequency-Aware Discrimination for Continual Test-Time Adaptation

TL;DR

This paper introduces a frequency-aware, self-adaptive expert framework for continual test-time adaptation, improving stability and efficiency in adapting to evolving domains while retaining prior knowledge.

Contribution

It proposes a dual-branch expert architecture and a frequency-aware domain discriminator to enhance continual adaptation and domain shift detection, along with a new benchmark for periodic domain changes.

Findings

Outperforms existing methods on classification and segmentation CTTA tasks.

Effectively reduces cross-domain interference and repetitive learning.

Demonstrates robustness and stability in dynamic domain environments.

Abstract

This paper focuses on the Continual Test-Time Adaptation (CTTA) task, aiming to enable an agent to continuously adapt to evolving target domains while retaining previously acquired domain knowledge for effective reuse when those domains reappear. Existing shared-parameter paradigms struggle to balance adaptation and forgetting, leading to decreased efficiency and stability. To address this, we propose a frequency-aware shared and self-adaptive expert framework, consisting of two key components: (i) a dual-branch expert architecture that extracts general features and dynamically models domain-specific representations, effectively reducing cross-domain interference and repetitive learning cost; and (ii) an online Frequency-aware Domain Discriminator (FDD), which leverages the robustness of low-frequency image signals for online domain shift detection, guiding dynamic allocation of expert resources for more stable and realistic adaptation. Additionally, we introduce a Continual Repeated Shifts (CRS) benchmark to simulate periodic domain changes for more realistic evaluation. Experimental results show that our method consistently outperforms existing approaches on both classification and segmentation CTTA tasks under standard and CRS settings, with ablations and visualizations confirming its effectiveness and robustness. Our code is available at https://github.com/ZJC25127/Domain-Self-Adaptive-CTTA.git.