Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EM

TL;DR

This paper introduces FreeTTA, a training-free test-time adaptation method that models test data distribution using online EM and zero-shot VLM predictions, significantly improving performance across diverse datasets.

Contribution

FreeTTA is the first method to explicitly model test data distribution in TTA without training or data storage, enhancing flexibility and prediction accuracy.

Findings

Achieves stable improvements over state-of-the-art methods.

Effective across 15 diverse datasets.

Enhances predictions by modeling test sample relationships.

Abstract

Vision-Language Models (VLMs) have become prominent in open-world image recognition for their strong generalization abilities. Yet, their effectiveness in practical applications is compromised by domain shifts and distributional changes, especially when test data distributions diverge from training data. Therefore, the paradigm of test-time adaptation (TTA) has emerged, enabling the use of online off-the-shelf data at test time, supporting independent sample predictions, and eliminating reliance on test annotations. Traditional TTA methods, however, often rely on costly training or optimization processes, or make unrealistic assumptions about accessing or storing historical training and test data. Instead, this study proposes FreeTTA, a training-free and universally available method that makes no assumptions, to enhance the flexibility of TTA. More importantly, FreeTTA is the first to explicitly model the test data distribution, enabling the use of intrinsic relationships among test samples to enhance predictions of individual samples without simultaneous access--a direction not previously explored. FreeTTA achieves these advantages by introducing an online EM algorithm that utilizes zero-shot predictions from VLMs as priors to iteratively compute the posterior probabilities of each online test sample and update parameters. Experiments demonstrate that FreeTTA achieves stable and significant improvements compared to state-of-the-art methods across 15 datasets in both cross-domain and out-of-distribution settings.