SSAM: Self-Supervised Association Modeling for Test-Time Adaption

TL;DR

SSAM introduces a self-supervised framework for test-time adaptation that refines image encoders dynamically, improving association modeling and handling distribution shifts without requiring explicit supervision.

Contribution

The paper proposes SSAM, a novel TTA method with dual-phase association learning, enabling dynamic encoder refinement and improved adaptation performance.

Findings

Outperforms state-of-the-art TTA methods on multiple benchmarks.

Maintains computational efficiency and architecture-agnostic design.

Effectively handles distribution shifts during test-time adaptation.

Abstract

Test-time adaption (TTA) has witnessed important progress in recent years, the prevailing methods typically first encode the image and the text and design strategies to model the association between them. Meanwhile, the image encoder is usually frozen due to the absence of explicit supervision in TTA scenarios. We identify a critical limitation in this paradigm: While test-time images often exhibit distribution shifts from training data, existing methods persistently freeze the image encoder due to the absence of explicit supervision during adaptation. This practice overlooks the image encoder's crucial role in bridging distribution shift between training and test. To address this challenge, we propose SSAM (Self-Supervised Association Modeling), a new TTA framework that enables dynamic encoder refinement through dual-phase association learning. Our method operates via two synergistic components: 1) Soft Prototype Estimation (SPE), which estimates probabilistic category associations to guide feature space reorganization, and 2) Prototype-anchored Image Reconstruction (PIR), enforcing encoder stability through cluster-conditional image feature reconstruction. Comprehensive experiments across diverse baseline methods and benchmarks demonstrate that SSAM can surpass state-of-the-art TTA baselines by a clear margin while maintaining computational efficiency. The framework's architecture-agnostic design and minimal hyperparameter dependence further enhance its practical applicability.