Dynamic Prompt Allocation and Tuning for Continual Test-Time Adaptation

TL;DR

This paper introduces a novel method called PAINT that uses dynamic, domain-specific prompts to improve continual test-time adaptation, effectively reducing catastrophic forgetting and handling unknown domains in evolving environments.

Contribution

The paper proposes a new prompt-based approach with a query mechanism for dynamic domain allocation, addressing limitations of existing methods in CTTA.

Findings

PAINT outperforms existing methods on benchmark datasets.

Dynamic prompt allocation reduces inter-domain interference.

Prompt tuning with mutual information maximization enhances adaptation.

Abstract

Continual test-time adaptation (CTTA) has recently emerged to adapt a pre-trained source model to continuously evolving target distributions, which accommodates the dynamic nature of real-world environments. To mitigate the risk of catastrophic forgetting in CTTA, existing methods typically incorporate explicit regularization terms to constrain the variation of model parameters. However, they cannot fundamentally resolve catastrophic forgetting because they rely on a single shared model to adapt across all target domains, which inevitably leads to severe inter-domain interference. In this paper, we introduce learnable domain-specific prompts that guide the model to adapt to corresponding target domains, thereby partially disentangling the parameter space of different domains. In the absence of domain identity for target samples, we propose a novel dynamic Prompt AllocatIon aNd Tuning (PAINT) method, which utilizes a query mechanism to dynamically determine whether the current samples come from a known domain or an unexplored one. For known domains, the corresponding domain-specific prompt is directly selected, while for previously unseen domains, a new prompt is allocated. Prompt tuning is subsequently performed using mutual information maximization along with structural regularization. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our PAINT method for CTTA. We have released our code at https://github.com/Cadezzyr/PAINT.