CAFA: Class-Aware Feature Alignment for Test-Time Adaptation

TL;DR

CAFA introduces a simple, hyper-parameter-free method for test-time adaptation that aligns features in a class-aware manner, improving model robustness across diverse datasets without needing source data during adaptation.

Contribution

The paper proposes a novel Class-Aware Feature Alignment (CAFA) loss that enhances test-time adaptation by promoting class-discriminative features without extra hyper-parameters.

Findings

Consistently outperforms existing baselines on 6 datasets.

Effectively mitigates distribution shifts during test-time adaptation.

Does not require source data or additional hyper-parameters.

Abstract

Despite recent advancements in deep learning, deep neural networks continue to suffer from performance degradation when applied to new data that differs from training data. Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time. TTA can be applied to pretrained networks without modifying their training procedures, enabling them to utilize a well-formed source distribution for adaptation. One possible approach is to align the representation space of test samples to the source distribution (\textit{i.e.,} feature alignment). However, performing feature alignment in TTA is especially challenging in that access to labeled source data is restricted during adaptation. That is, a model does not have a chance to learn test data in a class-discriminative manner, which was feasible in other adaptation tasks (\textit{e.g.,} unsupervised domain adaptation) via supervised losses on the source data. Based on this observation, we propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously 1) encourages a model to learn target representations in a class-discriminative manner and 2) effectively mitigates the distribution shifts at test time. Our method does not require any hyper-parameters or additional losses, which are required in previous approaches. We conduct extensive experiments on 6 different datasets and show our proposed method consistently outperforms existing baselines.