Optimizing Active Learning in Vision-Language Models via Parameter-Efficient Uncertainty Calibration

TL;DR

This paper presents a parameter-efficient active learning method for vision-language models that improves sample selection accuracy and efficiency by incorporating uncertainty calibration, outperforming existing techniques.

Contribution

It introduces a novel differentiable uncertainty calibration loss for active learning, enabling more effective and computationally efficient data sampling in large-scale vision-language models.

Findings

Outperforms complex feature-based sampling methods in accuracy

Achieves comparable or better results with less computational cost

Provides a comparative analysis of Prompt learning and LoRA in AL

Abstract

Active Learning (AL) has emerged as a powerful approach for minimizing labeling costs by selectively sampling the most informative data for neural network model development. Effective AL for large-scale vision-language models necessitates addressing challenges in uncertainty estimation and efficient sampling given the vast number of parameters involved. In this work, we introduce a novel parameter-efficient learning methodology that incorporates uncertainty calibration loss within the AL framework. We propose a differentiable loss function that promotes uncertainty calibration for effectively selecting fewer and most informative data samples for fine-tuning. Through extensive experiments across several datasets and vision backbones, we demonstrate that our solution can match and exceed the performance of complex feature-based sampling techniques while being computationally very efficient. Additionally, we investigate the efficacy of Prompt learning versus Low-rank adaptation (LoRA) in sample selection, providing a detailed comparative analysis of these methods in the context of efficient AL.