Towards Statistical Factuality Guarantee for Large Vision-Language Models

TL;DR

This paper introduces ConfLVLM, a conformal prediction framework that provides statistical guarantees to reduce hallucinations in large vision-language models, ensuring more reliable image-conditioned text generation.

Contribution

The paper proposes a novel conformal prediction-based method to verify and filter claims in LVLM outputs, offering finite-sample guarantees on factuality.

Findings

ConfLVLM reduces claim error rate from 87.8% to 10.0%.

Achieves 95.3% true positive rate in filtering unreliable claims.

Applicable to any black-box LVLM with various uncertainty measures.

Abstract

Advancements in Large Vision-Language Models (LVLMs) have demonstrated promising performance in a variety of vision-language tasks involving image-conditioned free-form text generation. However, growing concerns about hallucinations in LVLMs, where the generated text is inconsistent with the visual context, are becoming a major impediment to deploying these models in applications that demand guaranteed reliability. In this paper, we introduce a framework to address this challenge, ConfLVLM, which is grounded on conformal prediction to achieve finite-sample distribution-free statistical guarantees on the factuality of LVLM output. This framework treats an LVLM as a hypothesis generator, where each generated text detail (or claim) is considered an individual hypothesis. It then applies a statistical hypothesis testing procedure to verify each claim using efficient heuristic uncertainty measures to filter out unreliable claims before returning any responses to users. We conduct extensive experiments covering three representative application domains, including general scene understanding, medical radiology report generation, and document understanding. Remarkably, ConfLVLM reduces the error rate of claims generated by LLaVa-1.5 for scene descriptions from 87.8\% to 10.0\% by filtering out erroneous claims with a 95.3\% true positive rate. Our results further demonstrate that ConfLVLM is highly flexible, and can be applied to any black-box LVLMs paired with any uncertainty measure for any image-conditioned free-form text generation task while providing a rigorous guarantee on controlling the risk of hallucination.