Prototype-Based Test-Time Adaptation of Vision-Language Models

TL;DR

This paper introduces Prototype-Based Test-Time Adaptation (PTA), a highly efficient method for vision-language models that improves accuracy and speed by using class-specific prototypes without cache overhead.

Contribution

PTA is a novel TTA approach that adaptively updates class prototypes based on test samples, eliminating cache-related inefficiencies and achieving state-of-the-art results.

Findings

PTA improves CLIP accuracy from 65.64% to 69.38% on 10 benchmarks.

PTA retains 92% of CLIP's inference speed on large-scale datasets.

PTA outperforms cache-based TTA methods in both accuracy and efficiency.

Abstract

Test-time adaptation (TTA) has emerged as a promising paradigm for vision-language models (VLMs) to bridge the distribution gap between pre-training and test data. Recent works have focused on backpropagation-free TTA methods that rely on cache-based designs, but these introduce two key limitations. First, inference latency increases as the cache grows with the number of classes, leading to inefficiencies in large-scale settings. Second, suboptimal performance occurs when the cache contains insufficient or incorrect samples. In this paper, we present Prototype-Based Test-Time Adaptation (PTA), an efficient and effective TTA paradigm that uses a set of class-specific knowledge prototypes to accumulate knowledge from test samples. Particularly, knowledge prototypes are adaptively weighted based on the zero-shot class confidence of each test sample, incorporating the sample's visual features into the corresponding class-specific prototype. It is worth highlighting that the knowledge from past test samples is integrated and utilized solely in the prototypes, eliminating the overhead of cache population and retrieval that hinders the efficiency of existing TTA methods. This endows PTA with extremely high efficiency while achieving state-of-the-art performance on 15 image recognition benchmarks and 4 robust point cloud analysis benchmarks. For example, PTA improves CLIP's accuracy from 65.64% to 69.38% on 10 cross-domain benchmarks, while retaining 92% of CLIP's inference speed on large-scale ImageNet-1K. In contrast, the cache-based TDA achieves a lower accuracy of 67.97% and operates at only 50% of CLIP's inference speed.