PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models

TL;DR

This paper introduces PCoreSet, a novel sample selection strategy for active learning that leverages knowledge distillation from vision-language models, significantly improving efficiency and performance in data-scarce scenarios.

Contribution

It proposes Probabilistic CoreSet, a new selection method that exploits the structured prediction bias of vision-language models for active learning, enhancing knowledge transfer under limited labels.

Findings

PCoreSet outperforms existing selection strategies on 11 datasets.

ActiveKD with PCoreSet achieves up to +29.07% accuracy on ImageNet.

The method ranks first in 87.7% of evaluated settings.

Abstract

Knowledge distillation (KD) is a widely used framework for training compact, task-specific models by transferring the knowledge from teacher models. However, its application to active learning (AL), which aims to minimize annotation costs through iterative sample selection, remains underexplored. This gap stems from the fact that KD typically assumes access to sufficient labeled data, whereas AL operates in data-scarce scenarios where task-specific teacher models are often unavailable. In this paper, we first introduce ActiveKD, a framework that integrates AL with KD by leveraging the zero- and few-shot capabilities of large vision-language models (VLMs). A key aspect of ActiveKD is the structured prediction bias of VLMs-i.e., their predictions form clusters in the probability space. We regard this structure as an inductive bias of the teacher model, capturing generalizable output patterns beneficial to student learning. To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection strategy that maximizes coverage in the probability space rather than the feature space. PCoreSet strategically selects probabilistically diverse unlabeled samples, facilitating more efficient transfer of teacher knowledge under limited annotation budgets. Extensive evaluations on 11 datasets show that ActiveKD consistently improves performance across selection methods (e.g., +29.07% on ImageNet, averaged over methods). Under ActiveKD, PCoreSet ranks first in 64/73 settings (approximately 87.7%) across 5 student and 3 teacher networks, always achieving the best performance except for first 2 AL rounds. Our code is available at https://github.com/erjui/PCoreSet.