Generative Score Inference for Multimodal Data

TL;DR

The paper introduces Generative Score Inference (GSI), a flexible framework for uncertainty quantification in multimodal data, leveraging deep generative models to improve reliability across diverse tasks.

Contribution

GSI provides a novel, generalizable inference method that constructs valid prediction and confidence sets using synthetic samples, overcoming limitations of existing approaches.

Findings

GSI achieves state-of-the-art hallucination detection in language models.

GSI provides robust uncertainty estimates in image captioning.

Performance improves with higher quality generative models.

Abstract

Accurate uncertainty quantification is crucial for making reliable decisions in various supervised learning scenarios, particularly when dealing with complex, multimodal data such as images and text. Current approaches often face notable limitations, including rigid assumptions and limited generalizability, constraining their effectiveness across diverse supervised learning tasks. To overcome these limitations, we introduce Generative Score Inference (GSI), a flexible inference framework capable of constructing statistically valid and informative prediction and confidence sets across a wide range of multimodal learning problems. GSI utilizes synthetic samples generated by deep generative models to approximate conditional score distributions, facilitating precise uncertainty quantification without imposing restrictive assumptions about the data or tasks. We empirically validate GSI's capabilities through two representative scenarios: hallucination detection in large language models and uncertainty estimation in image captioning. Our method achieves state-of-the-art performance in hallucination detection and robust predictive uncertainty in image captioning, and its performance is positively influenced by the quality of the underlying generative model. These findings underscore the potential of GSI as a versatile inference framework, significantly enhancing uncertainty quantification and trustworthiness in multimodal learning.