UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models

TL;DR

This paper introduces UP-DP, an unsupervised prompt learning method that enhances vision-language models for data pre-selection, leading to better representation and significant performance improvements across multiple datasets.

Contribution

It is the first to incorporate unsupervised prompt learning into vision-language models for data pre-selection, improving dataset representation and generalizability.

Findings

Achieves up to 20% performance gain over state-of-the-art methods.

Prompts learned from one dataset generalize well to others.

Joint vision-text features outperform visual-only features in data pre-selection.

Abstract

In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks with a limited annotation budget. Previous approaches to data pre-selection relied solely on visual features extracted from foundation models, such as CLIP and BLIP-2, but largely ignored the powerfulness of text features. In this work, we argue that, with proper design, the joint feature space of both vision and text can yield a better representation for data pre-selection. To this end, we introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision-language models, like BLIP-2, for data pre-selection. Specifically, with the BLIP-2 parameters frozen, we train text prompts to extract the joint features with improved representation, ensuring a diverse cluster structure that covers the entire dataset. We extensively compare our method with the state-of-the-art using seven benchmark datasets in different settings, achieving up to a performance gain of 20%. Interestingly, the prompts learned from one dataset demonstrate significant generalizability and can be applied directly to enhance the feature extraction of BLIP-2 from other datasets. To the best of our knowledge, UP-DP is the first work to incorporate unsupervised prompt learning in a vision-language model for data pre-selection.