BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptation

TL;DR

BayesAdapter enhances uncertainty estimation in CLIP few-shot adaptation by leveraging Bayesian inference, resulting in better calibration and selective classification, addressing a gap in the reliability of existing adapters.

Contribution

It introduces BayesAdapter, a Bayesian approach for CLIP adapters that improves uncertainty estimation without sacrificing discriminative performance.

Findings

BayesAdapter provides well-calibrated uncertainty estimates.

It outperforms existing adapters in selective classification tasks.

The method demonstrates robustness across various downstream tasks.

Abstract

The emergence of large pre-trained vision-language models (VLMs) represents a paradigm shift in machine learning, with unprecedented results in a broad span of visual recognition tasks. CLIP, one of the most popular VLMs, has exhibited remarkable zero-shot and transfer learning capabilities in classification. To transfer CLIP to downstream tasks, adapters constitute a parameter-efficient approach that avoids backpropagation through the large model (unlike related prompt learning methods). However, CLIP adapters have been developed to target discriminative performance, and the quality of their uncertainty estimates has been overlooked. In this work we show that the discriminative performance of state-of-the-art CLIP adapters does not always correlate with their uncertainty estimation capabilities, which are essential for a safe deployment in real-world scenarios. We also demonstrate that one of such adapters is obtained through MAP inference from a more general probabilistic framework. Based on this observation we introduce BayesAdapter, which leverages Bayesian inference to estimate a full probability distribution instead of a single point, better capturing the variability inherent in the parameter space. In a comprehensive empirical evaluation we show that our approach obtains high quality uncertainty estimates in the predictions, standing out in calibration and selective classification. Our code will be publicly available upon acceptance of the paper.