Unsupervised Prototype Adapter for Vision-Language Models

TL;DR

This paper introduces an unsupervised fine-tuning method called Unsupervised Prototype Adapter (UP-Adapter) for vision-language models like CLIP, enabling effective downstream recognition without annotated data.

Contribution

The paper proposes a novel unsupervised approach to adapt vision-language models using automatically selected samples and class prototypes, eliminating the need for labeled data.

Findings

Outperforms 8-shot CoOp and Tip-Adapter in image recognition tasks.

Achieves superior results on domain generalization benchmarks.

Demonstrates effectiveness without requiring annotated datasets.

Abstract

Recently, large-scale pre-trained vision-language models (e.g. CLIP and ALIGN) have demonstrated remarkable effectiveness in acquiring transferable visual representations. To leverage the valuable knowledge encoded within these models for downstream tasks, several fine-tuning approaches, including prompt tuning methods and adapter-based methods, have been developed to adapt vision-language models effectively with supervision. However, these methods rely on the availability of annotated samples, which can be labor-intensive and time-consuming to acquire, thus limiting scalability. To address this issue, in this work, we design an unsupervised fine-tuning approach for vision-language models called Unsupervised Prototype Adapter (UP-Adapter). Specifically, for the unannotated target datasets, we leverage the text-image aligning capability of CLIP to automatically select the most confident samples for each class. Utilizing these selected samples, we generate class prototypes, which serve as the initialization for the learnable prototype model. After fine-tuning, the prototype model prediction is combined with the original CLIP's prediction by a residual connection to perform downstream recognition tasks. Our extensive experimental results on image recognition and domain generalization show that the proposed unsupervised method outperforms 8-shot CoOp, 8-shot Tip-Adapter, and also the state-of-the-art UPL method by large margins.