The Consistency Hypothesis in Uncertainty Quantification for Large Language Models

TL;DR

This paper investigates the assumption that consistency in LLM outputs can serve as a reliable measure of confidence, proposing new statistical tests and data-free methods that improve uncertainty estimation across various tasks.

Contribution

It formalizes the consistency hypothesis, introduces statistical tests and metrics for evaluation, and develops data-free UQ methods that leverage output similarities for better confidence estimation.

Findings

The consistency hypothesis is prevalent across multiple tasks and datasets.

The 'Sim-Any' hypothesis is particularly effective for confidence estimation.

Data-free methods based on output similarity can outperform existing baselines.

Abstract

Estimating the confidence of large language model (LLM) outputs is essential for real-world applications requiring high user trust. Black-box uncertainty quantification (UQ) methods, relying solely on model API access, have gained popularity due to their practical benefits. In this paper, we examine the implicit assumption behind several UQ methods, which use generation consistency as a proxy for confidence, an idea we formalize as the consistency hypothesis. We introduce three mathematical statements with corresponding statistical tests to capture variations of this hypothesis and metrics to evaluate LLM output conformity across tasks. Our empirical investigation, spanning 8 benchmark datasets and 3 tasks (question answering, text summarization, and text-to-SQL), highlights the prevalence of the hypothesis under different settings. Among the statements, we highlight the `Sim-Any' hypothesis as the most actionable, and demonstrate how it can be leveraged by proposing data-free black-box UQ methods that aggregate similarities between generations for confidence estimation. These approaches can outperform the closest baselines, showcasing the practical value of the empirically observed consistency hypothesis.