CLIPArTT: Adaptation of CLIP to New Domains at Test Time

TL;DR

CLIPArTT introduces a test-time adaptation method for CLIP that constructs automatic text prompts during inference, improving robustness across various datasets without additional training or model modifications.

Contribution

The paper presents a novel, minimally invasive test-time adaptation technique for CLIP that uses pseudo labels from aggregated class prompts, and standardizes TTA benchmarks for vision-language models.

Findings

Enhances CLIP performance on corrupted and synthetic datasets

Does not require additional training or model modifications

Improves robustness across diverse datasets

Abstract

Pre-trained vision-language models (VLMs), exemplified by CLIP, demonstrate remarkable adaptability across zero-shot classification tasks without additional training. However, their performance diminishes in the presence of domain shifts. In this study, we introduce CLIP Adaptation duRing Test-Time (CLIPArTT), a fully test-time adaptation (TTA) approach for CLIP, which involves automatic text prompts construction during inference for their use as text supervision. Our method employs a unique, minimally invasive text prompt tuning process, wherein multiple predicted classes are aggregated into a single new text prompt, used as \emph{pseudo label} to re-classify inputs in a transductive manner. Additionally, we pioneer the standardization of TTA benchmarks (e.g., TENT) in the realm of VLMs. Our findings demonstrate that, without requiring additional transformations nor new trainable modules, CLIPArTT enhances performance dynamically across non-corrupted datasets such as CIFAR-100, corrupted datasets like CIFAR-100-C and ImageNet-C, alongside synthetic datasets such as VisDA-C. This research underscores the potential for improving VLMs' adaptability through novel test-time strategies, offering insights for robust performance across varied datasets and environments. The code can be found at: https://github.com/dosowiechi/CLIPArTT.git