Dual Strategies for Test-Time Adaptation

TL;DR

DualTTA enhances test-time adaptation by dynamically selecting and leveraging diverse test samples, improving robustness under distribution shifts through entropy-based strategies and reliability criteria.

Contribution

The paper introduces DualTTA, a novel framework that adaptively distinguishes reliable and unreliable test samples for more effective model updates under distribution shifts.

Findings

DualTTA outperforms existing TTA methods on various benchmarks.

It effectively separates reliable and unreliable samples using a new reliability criterion.

Theoretical analysis confirms tighter sample selection improves adaptation.

Abstract

Conventional test-time adaptation (TTA) approaches typically adapt the model using only a small fraction of test samples, often those with low-entropy predictions, thereby failing to fully leverage the available information in the test distribution. This paper introduces DualTTA, a novel framework that improves performance under distribution shifts by utilizing a larger and more diverse set of test samples. DualTTA identifies two distinct groups: one where the model's predictions are likely consistent with the underlying semantics, and another where predictions are likely incorrect. For the first group, it minimizes prediction entropy to reinforce reliable decisions; for the second, it maximizes entropy to suppress overconfident errors and unlearn spurious behavior. These groups are adaptively selected using a new reliability criterion that measures prediction stability under both semantic-preserving and semantic-altering transformations, addressing the limitations of purely entropy-based selection. We further provide theoretical analysis and empirical justification showing that our approach enables a tighter separation between reliable and unreliable samples, in the context of their suitability for adaptation, leading to provably more effective model updates.