Tuning Vision-Language Models with Candidate Labels by Prompt Alignment

TL;DR

This paper explores prompt learning for vision-language models using candidate labels, proposing a framework that disambiguates labels and improves robustness, especially when label ambiguity is high.

Contribution

It introduces the first study on prompt learning with candidate labels for VLMs and proposes a framework that leverages prior knowledge to enhance performance.

Findings

Prompt learning outperforms other fine-tuning methods with candidate labels.

Performance declines as label ambiguity increases.

The proposed framework effectively disambiguates labels and improves robustness.

Abstract

Vision-language models (VLMs) can learn high-quality representations from a large-scale training dataset of image-text pairs. Prompt learning is a popular approach to fine-tuning VLM to adapt them to downstream tasks. Despite the satisfying performance, a major limitation of prompt learning is the demand for labelled data. In real-world scenarios, we may only obtain candidate labels (where the true label is included) instead of the true labels due to data privacy or sensitivity issues. In this paper, we provide the first study on prompt learning with candidate labels for VLMs. We empirically demonstrate that prompt learning is more advantageous than other fine-tuning methods, for handling candidate labels. Nonetheless, its performance drops when the label ambiguity increases. In order to improve its robustness, we propose a simple yet effective framework that better leverages the prior knowledge of VLMs to guide the learning process with candidate labels. Specifically, our framework disambiguates candidate labels by aligning the model output with the mixed class posterior jointly predicted by both the learnable and the handcrafted prompt. Besides, our framework can be equipped with various off-the-shelf training objectives for learning with candidate labels to further improve their performance. Extensive experiments demonstrate the effectiveness of our proposed framework.