Unsupervised Prompt Learning for Vision-Language Models

TL;DR

This paper introduces an unsupervised prompt learning method for vision-language models like CLIP, eliminating the need for labeled data and prompt engineering, and achieving superior transfer performance across multiple datasets.

Contribution

It is the first to incorporate unsupervised learning into prompt learning, enhancing CLIP's transfer capabilities without requiring labeled target data.

Findings

Outperforms original CLIP with prompt engineering on ImageNet and 10 other datasets.

Competitive with 8-shot CoOp and TIP-Adapter methods.

Demonstrates effectiveness of unsupervised prompt learning in vision-language models.

Abstract

Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explore a different scenario, in which the labels of the target datasets are unprovided, and we present an unsupervised prompt learning (UPL) approach to avoid prompt engineering while simultaneously improving transfer performance of CLIP-like vision-language models. As far as we know, UPL is the first work to introduce unsupervised learning into prompt learning. Experimentally, our UPL outperforms original CLIP with prompt engineering on ImageNet as well as other 10 datasets. An enhanced version of UPL is even competitive with the 8-shot CoOp and the 8-shot TIP-Adapter on most datasets. Code and models are available at https://github.com/tonyhuang2022/UPL.