Realistic Test-Time Adaptation of Vision-Language Models

TL;DR

This paper evaluates the limitations of current test-time adaptation methods for vision-language models under realistic scenarios and introduces StatA, a new method that maintains robustness across variable class distributions during deployment.

Contribution

The paper proposes a realistic evaluation framework for TTA of VLMs and introduces StatA, a versatile adaptation method with a novel regularization to preserve initial knowledge.

Findings

Current TTA methods compromise zero-shot robustness in realistic scenarios.

StatA effectively handles variable class distributions during test time.

StatA maintains model robustness across diverse deployment conditions.

Abstract

The zero-shot capabilities of Vision-Language Models (VLMs) have been widely leveraged to improve predictive performance. However, previous works on transductive or test-time adaptation (TTA) often make strong assumptions about the data distribution, such as the presence of all classes. Our work challenges these favorable deployment scenarios, and introduces a more realistic evaluation framework, including: (i) a variable number of effective classes for adaptation within a single batch, and (ii) non-i.i.d. batches of test samples in online adaptation settings. We provide comprehensive evaluations, comparisons, and ablation studies that demonstrate how current transductive or TTA methods for VLMs systematically compromise the models' initial zero-shot robustness across various realistic scenarios, favoring performance gains under advantageous assumptions about the test samples' distributions. Furthermore, we introduce StatA, a versatile method that could handle a wide range of deployment scenarios, including those with a variable number of effective classes at test time. Our approach incorporates a novel regularization term designed specifically for VLMs, which acts as a statistical anchor preserving the initial text-encoder knowledge, particularly in low-data regimes. Code available at https://github.com/MaxZanella/StatA.