Active Prompt Learning in Vision Language Models

TL;DR

This paper introduces PCB, a novel active learning framework for Vision Language Models that addresses class imbalance issues and leverages VLM knowledge to improve task adaptation, outperforming traditional methods.

Contribution

The paper proposes a new active learning method, PCB, tailored for VLMs, which effectively balances classes and utilizes model knowledge to enhance performance.

Findings

PCB outperforms conventional active learning methods.

Applying standard active learning may degrade VLM performance.

Knowledge-guided sampling improves label efficiency.

Abstract

Pre-trained Vision Language Models (VLMs) have demonstrated notable progress in various zero-shot tasks, such as classification and retrieval. Despite their performance, because improving performance on new tasks requires task-specific knowledge, their adaptation is essential. While labels are needed for the adaptation, acquiring them is typically expensive. To overcome this challenge, active learning, a method of achieving a high performance by obtaining labels for a small number of samples from experts, has been studied. Active learning primarily focuses on selecting unlabeled samples for labeling and leveraging them to train models. In this study, we pose the question, "how can the pre-trained VLMs be adapted under the active learning framework?" In response to this inquiry, we observe that (1) simply applying a conventional active learning framework to pre-trained VLMs even may degrade performance compared to random selection because of the class imbalance in labeling candidates, and (2) the knowledge of VLMs can provide hints for achieving the balance before labeling. Based on these observations, we devise a novel active learning framework for VLMs, denoted as PCB. To assess the effectiveness of our approach, we conduct experiments on seven different real-world datasets, and the results demonstrate that PCB surpasses conventional active learning and random sampling methods. Code will be available in https://github.com/kaist-dmlab/pcb .