Neural Collapse in Test-Time Adaptation

TL;DR

This paper introduces NCTTA, a novel method for test-time adaptation that aligns features with classifier weights to improve robustness against domain shifts, supported by theoretical insights from neural collapse phenomena.

Contribution

It extends neural collapse to sample-wise analysis, identifies the cause of performance degradation, and proposes a new alignment method with hybrid targets for better test-time adaptation.

Findings

NCTTA outperforms Tent by 14.52% on ImageNet-C.

Sample-wise alignment collapse (NC3+) explains performance degradation.

Hybrid targets improve pseudo-label reliability under domain shifts.

Abstract

Test-Time Adaptation (TTA) enhances model robustness to out-of-distribution (OOD) data by updating the model online during inference, yet existing methods lack theoretical insights into the fundamental causes of performance degradation under domain shifts. Recently, Neural Collapse (NC) has been proposed as an emergent geometric property of deep neural networks (DNNs), providing valuable insights for TTA. In this work, we extend NC to the sample-wise level and discover a novel phenomenon termed Sample-wise Alignment Collapse (NC3+), demonstrating that a sample's feature embedding, obtained by a trained model, aligns closely with the corresponding classifier weight. Building on NC3+, we identify that the performance degradation stems from sample-wise misalignment in adaptation which exacerbates under larger distribution shifts. This indicates the necessity of realigning the feature embeddings with their corresponding classifier weights. However, the misalignment makes pseudo-labels unreliable under domain shifts. To address this challenge, we propose NCTTA, a novel feature-classifier alignment method with hybrid targets to mitigate the impact of unreliable pseudo-labels, which blends geometric proximity with predictive confidence. Extensive experiments demonstrate the effectiveness of NCTTA in enhancing robustness to domain shifts. For example, NCTTA outperforms Tent by 14.52% on ImageNet-C. Project page is publicly available at https://github.com/Cevaaa/NCTTA.