Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data

TL;DR

This paper introduces CPL, a novel method for fine-tuning vision-language models using candidate pseudolabels generated through a confidence-based selection process, improving performance with unlabeled data.

Contribution

We propose a candidate pseudolabel learning approach that refines pseudolabels via intra- and inter-instance selection, enhancing unlabeled data utilization in VLM fine-tuning.

Findings

Significant performance gains on nine benchmark datasets.

Effective handling of low zero-shot performance scenarios.

Improved class-balanced instance selection.

Abstract

Fine-tuning vision-language models (VLMs) with abundant unlabeled data recently has attracted increasing attention. Existing methods that resort to the pseudolabeling strategy would suffer from heavily incorrect hard pseudolabels when VLMs exhibit low zero-shot performance in downstream tasks. To alleviate this issue, we propose a Candidate Pseudolabel Learning method, termed CPL, to fine-tune VLMs with suitable candidate pseudolabels of unlabeled data in downstream tasks. The core of our method lies in the generation strategy of candidate pseudolabels, which progressively generates refined candidate pseudolabels by both intra- and inter-instance label selection, based on a confidence score matrix for all unlabeled data. This strategy can result in better performance in true label inclusion and class-balanced instance selection. In this way, we can directly apply existing loss functions to learn with generated candidate psueudolabels. Extensive experiments on nine benchmark datasets with three learning paradigms demonstrate the effectiveness of our method. Our code can be found at https://github.com/vanillaer/CPL-ICML2024.