Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere

TL;DR

This paper introduces AsymVLM, a method to model the asymmetric uncertainty in pre-trained vision-language models by creating probabilistic embeddings on the unit hypersphere, improving uncertainty quantification.

Contribution

It proposes a novel approach to capture asymmetric uncertainty in VLMs by constructing probabilistic embeddings on the hypersphere, addressing limitations of previous methods.

Findings

Probabilistic embeddings improve uncertainty estimation.

AsymVLM outperforms existing post-hoc adaptation methods.

Ablation studies confirm the asymmetry in data uncertainty structures.

Abstract

Vision-language models (VLMs) as foundation models have significantly enhanced performance across a wide range of visual and textual tasks, without requiring large-scale training from scratch for downstream tasks. However, these deterministic VLMs fail to capture the inherent ambiguity and uncertainty in natural language and visual data. Recent probabilistic post-hoc adaptation methods address this by mapping deterministic embeddings onto probability distributions; however, existing approaches do not account for the asymmetric uncertainty structure of the modalities, and the constraint that meaningful deterministic embeddings reside on a unit hypersphere, potentially leading to suboptimal performance. In this paper, we address the asymmetric uncertainty structure inherent in textual and visual data, and propose AsymVLM to build probabilistic embeddings from pre-trained VLMs on the unit hypersphere, enabling uncertainty quantification. We validate the effectiveness of the probabilistic embeddings on established benchmarks, and present comprehensive ablation studies demonstrating the inherent nature of asymmetry in the uncertainty structure of textual and visual data.