WATT: Weight Average Test-Time Adaptation of CLIP

TL;DR

WATT enhances CLIP's zero-shot image classification by employing test-time adaptation with weight averaging and text ensemble strategies, significantly improving performance across various domain-shifted datasets without additional training.

Contribution

This paper introduces WATT, a novel test-time adaptation method for CLIP that uses pseudo labels, weight averaging, and text ensemble strategies to improve robustness without extra training modules.

Findings

WATT improves CLIP's performance on multiple domain-shifted datasets.

The method operates effectively with just a single image per test case.

WATT does not require additional model training or transformations.

Abstract

Vision-Language Models (VLMs) such as CLIP have yielded unprecedented performance for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. In response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, a pioneering approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing overall test performance by aggregating diverse textual cues. Our findings underscore the efficacy of WATT in enhancing performance across diverse datasets, including CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, VisDA-C, and several other challenging datasets, effectively covering a wide range of domain shifts. Notably, these enhancements are achieved without necessitating additional model transformations or trainable modules. Moreover, compared to other Test-Time Adaptation methods, our approach can operate effectively with just a single image. Highlighting the potential of innovative test-time strategies, this research emphasizes their role in fortifying the adaptability of VLMs. The implementation is available at: \url{https://github.com/Mehrdad-Noori/WATT.git}.