Active Prompt Learning with Vision-Language Model Priors

TL;DR

This paper presents an active prompt learning framework for vision-language models that uses class-guided clustering and adaptive thresholds to efficiently select data, improving accuracy with fewer labeled samples.

Contribution

It introduces a novel class-guided clustering and adaptive threshold-based active learning method tailored for vision-language models, enhancing data efficiency.

Findings

Outperforms existing baselines across seven datasets.

Reduces labeling costs while maintaining high accuracy.

Effectively leverages pre-trained encoders for data selection.

Abstract

Vision-language models (VLMs) have demonstrated remarkable zero-shot performance across various classification tasks. Nonetheless, their reliance on hand-crafted text prompts for each task hinders efficient adaptation to new tasks. While prompt learning offers a promising solution, most studies focus on maximizing the utilization of given few-shot labeled datasets, often overlooking the potential of careful data selection strategies, which enable higher accuracy with fewer labeled data. This motivates us to study a budget-efficient active prompt learning framework. Specifically, we introduce a class-guided clustering that leverages the pre-trained image and text encoders of VLMs, thereby enabling our cluster-balanced acquisition function from the initial round of active learning. Furthermore, considering the substantial class-wise variance in confidence exhibited by VLMs, we propose a budget-saving selective querying based on adaptive class-wise thresholds. Extensive experiments in active learning scenarios across seven datasets demonstrate that our method outperforms existing baselines.