Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matching

TL;DR

This paper introduces REPVLM, a method that quantifies epistemic uncertainty in vision-language models by estimating embedding densities on hyperspherical manifolds, improving out-of-distribution detection.

Contribution

The paper proposes a novel approach using Riemannian Flow Matching to compute embedding densities for uncertainty quantification in VLMs, addressing a key limitation of deterministic models.

Findings

REPVLM achieves near-perfect correlation between uncertainty and prediction error.

Outperforms existing baselines in uncertainty quantification tasks.

Provides scalable metrics for out-of-distribution detection and data curation.

Abstract

Vision-Language Models (VLMs) are typically deterministic in nature and lack intrinsic mechanisms to quantify epistemic uncertainty, which reflects the model's lack of knowledge or ignorance of its own representations. We theoretically motivate negative log-density of an embedding as a proxy for the epistemic uncertainty, where low-density regions signify model ignorance. The proposed method REPVLM computes the probability density on the hyperspherical manifold of the VLM embeddings using Riemannian Flow Matching. We empirically demonstrate that REPVLM achieves near-perfect correlation between uncertainty and prediction error, significantly outperforming existing baselines. Beyond classification, we also demonstrate that the model also provides a scalable metric for out-of-distribution detection and automated data curation.