Open-World Test-Time Adaptation with Hierarchical Feature Aggregation and Attention Affine

TL;DR

This paper introduces a novel test-time adaptation framework combining hierarchical feature aggregation and attention mechanisms to improve model robustness and accuracy in open-world, out-of-distribution scenarios.

Contribution

The paper proposes a Hierarchical Ladder Network and an Attention Affine Network for enhanced OOD detection and domain adaptation during test time.

Findings

Significant improvement in classification accuracy on benchmark datasets.

Effective OOD detection through hierarchical feature aggregation.

Robust adaptation under domain shift with attention-based refinement.

Abstract

Test-time adaptation (TTA) refers to adjusting the model during the testing phase to cope with changes in sample distribution and enhance the model's adaptability to new environments. In real-world scenarios, models often encounter samples from unseen (out-of-distribution, OOD) categories. Misclassifying these as known (in-distribution, ID) classes not only degrades predictive accuracy but can also impair the adaptation process, leading to further errors on subsequent ID samples. Many existing TTA methods suffer substantial performance drops under such conditions. To address this challenge, we propose a Hierarchical Ladder Network that extracts OOD features from class tokens aggregated across all Transformer layers. OOD detection performance is enhanced by combining the original model prediction with the output of the Hierarchical Ladder Network (HLN) via weighted probability fusion. To improve robustness under domain shift, we further introduce an Attention Affine Network (AAN) that adaptively refines the self-attention mechanism conditioned on the token information to better adapt to domain drift, thereby improving the classification performance of the model on datasets with domain shift. Additionally, a weighted entropy mechanism is employed to dynamically suppress the influence of low-confidence samples during adaptation. Experimental results on benchmark datasets show that our method significantly improves the performance on the most widely used classification datasets.