ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models

TL;DR

ProbVLM introduces a probabilistic approach to estimate uncertainty in embeddings of pre-trained vision-language models, improving retrieval, active learning, and model selection without large datasets or additional training.

Contribution

It presents ProbVLM, a novel post-hoc probabilistic adapter that captures embedding uncertainties in VLMs, enhancing their interpretability and downstream task performance.

Findings

ProbVLM outperforms existing methods in uncertainty estimation across four datasets.

Uncertainty estimates improve retrieval accuracy and model selection.

Visualization of embedding distributions is enabled using a latent diffusion model.

Abstract

Large-scale vision-language models (VLMs) like CLIP successfully find correspondences between images and text. Through the standard deterministic mapping process, an image or a text sample is mapped to a single vector in the embedding space. This is problematic: as multiple samples (images or text) can abstract the same concept in the physical world, deterministic embeddings do not reflect the inherent ambiguity in the embedding space. We propose ProbVLM, a probabilistic adapter that estimates probability distributions for the embeddings of pre-trained VLMs via inter/intra-modal alignment in a post-hoc manner without needing large-scale datasets or computing. On four challenging datasets, i.e., COCO, Flickr, CUB, and Oxford-flowers, we estimate the multi-modal embedding uncertainties for two VLMs, i.e., CLIP and BLIP, quantify the calibration of embedding uncertainties in retrieval tasks and show that ProbVLM outperforms other methods. Furthermore, we propose active learning and model selection as two real-world downstream tasks for VLMs and show that the estimated uncertainty aids both tasks. Lastly, we present a novel technique for visualizing the embedding distributions using a large-scale pre-trained latent diffusion model. Code is available at https://github.com/ExplainableML/ProbVLM.