Uncover and Unlearn Nuisances: Agnostic Fully Test-Time Adaptation

TL;DR

This paper introduces Agnostic Fully Test-Time Adaptation (AFTTA), a novel method that enables models to adapt to unforeseen domain shifts during testing by uncovering and unlearning nuisances without source data.

Contribution

AFTTA is the first approach to explicitly address agnostic domain shifts in test-time adaptation using an uncover-and-unlearn strategy with mutual information guidance.

Findings

Outperforms existing FTTA methods on corruption and style shift tasks.

Effectively unlearns nuisances to improve generalization.

Enhances model robustness without source data access.

Abstract

Fully Test-Time Adaptation (FTTA) addresses domain shifts without access to source data and training protocols of the pre-trained models. Traditional strategies that align source and target feature distributions are infeasible in FTTA due to the absence of training data and unpredictable target domains. In this work, we exploit a dual perspective on FTTA, and propose Agnostic FTTA (AFTTA) as a novel formulation that enables the usage of off-the-shelf domain transformations during test-time to enable direct generalization to unforeseeable target data. To address this, we develop an uncover-and-unlearn approach. First, we uncover potential unwanted shifts between source and target domains by simulating them through predefined mappings and consider them as nuisances. Then, during test-time prediction, the model is enforced to unlearn these nuisances by regularizing the consequent shifts in latent representations and label predictions. Specifically, a mutual information-based criterion is devised and applied to guide nuisances unlearning in the feature space and encourage confident and consistent prediction in label space. Our proposed approach explicitly addresses agnostic domain shifts, enabling superior model generalization under FTTA constraints. Extensive experiments on various tasks, involving corruption and style shifts, demonstrate that our method consistently outperforms existing approaches.