Conformal Cross-Modal Active Learning

TL;DR

This paper presents CCMA, a novel active learning framework that leverages vision-language models to improve data efficiency by using conformal calibration and multimodal uncertainty estimates for sample selection.

Contribution

Introducing Conformal Cross-Modal Acquisition (CCMA), a new active learning method that combines multimodal uncertainty estimation with diversity strategies using pretrained vision-language models.

Findings

CCMA outperforms existing active learning methods on multiple benchmarks.

The approach effectively reduces annotation costs while maintaining high accuracy.

Multimodal conformal scoring enhances sample selection quality.

Abstract

Foundation models for vision have transformed visual recognition with powerful pretrained representations and strong zero-shot capabilities, yet their potential for data-efficient learning remains largely untapped. Active Learning (AL) aims to minimize annotation costs by strategically selecting the most informative samples for labeling, but existing methods largely overlook the rich multimodal knowledge embedded in modern vision-language models (VLMs). We introduce Conformal Cross-Modal Acquisition (CCMA), a novel AL framework that bridges vision and language modalities through a teacher-student architecture. CCMA employs a pretrained VLM as a teacher to provide semantically grounded uncertainty estimates, conformally calibrated to guide sample selection for a vision-only student model. By integrating multimodal conformal scoring with diversity-aware selection strategies, CCMA achieves superior data efficiency across multiple benchmarks. Our approach consistently outperforms state-of-the-art AL baselines, demonstrating clear advantages over methods relying solely on uncertainty or diversity metrics.