The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models

TL;DR

This paper introduces TTA-VLM, a comprehensive benchmark for evaluating test-time adaptation methods on vision-language models, revealing limited gains and trade-offs in trustworthiness, to foster more reliable future strategies.

Contribution

The paper presents TTA-VLM, a unified benchmark with diverse evaluation metrics, extending analysis beyond CLIP to SigLIP and training-time tuning methods for fair comparison.

Findings

Existing TTA methods show limited improvements.

Poor synergy between TTA and training-time fine-tuning.

Accuracy gains often reduce model trustworthiness.

Abstract

Test-time adaptation (TTA) methods have gained significant attention for enhancing the performance of vision-language models (VLMs) such as CLIP during inference, without requiring additional labeled data. However, current TTA researches generally suffer from major limitations such as duplication of baseline results, limited evaluation metrics, inconsistent experimental settings, and insufficient analysis. These problems hinder fair comparisons between TTA methods and make it difficult to assess their practical strengths and weaknesses. To address these challenges, we introduce TTA-VLM, a comprehensive benchmark for evaluating TTA methods on VLMs. Our benchmark implements 8 episodic TTA and 7 online TTA methods within a unified and reproducible framework, and evaluates them across 15 widely used datasets. Unlike prior studies focused solely on CLIP, we extend the evaluation to SigLIP--a model trained with a Sigmoid loss--and include training-time tuning methods such as CoOp, MaPLe, and TeCoA to assess generality. Beyond classification accuracy, TTA-VLM incorporates various evaluation metrics, including robustness, calibration, out-of-distribution detection, and stability, enabling a more holistic assessment of TTA methods. Through extensive experiments, we find that 1) existing TTA methods produce limited gains compared to the previous pioneering work; 2) current TTA methods exhibit poor collaboration with training-time fine-tuning methods; 3) accuracy gains frequently come at the cost of reduced model trustworthiness. We release TTA-VLM to provide fair comparison and comprehensive evaluation of TTA methods for VLMs, and we hope it encourages the community to develop more reliable and generalizable TTA strategies.