Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities

TL;DR

This paper introduces prompt-based techniques using imprecise probabilities to better capture and quantify higher-order uncertainty in large language models, addressing limitations of classical probabilistic methods.

Contribution

It proposes a novel framework for eliciting and quantifying both first- and second-order uncertainties in LLMs through prompt design and post-processing, grounded in imprecise probabilities.

Findings

Improved uncertainty reporting fidelity in LLMs.

Effective elicitation of higher-order uncertainty.

Enhanced decision-making support from LLMs.

Abstract

Despite the growing demand for eliciting uncertainty from large language models (LLMs), empirical evidence suggests that LLM behavior is not always adequately captured by the elicitation techniques developed under the classical probabilistic uncertainty framework. This mismatch leads to systematic failure modes, particularly in settings that involve ambiguous question-answering, in-context learning, and self-reflection. To address this, we propose novel prompt-based uncertainty elicitation techniques grounded in \emph{imprecise probabilities}, a principled framework for repesenting and eliciting higher-order uncertainty. Here, first-order uncertainty captures uncertainty over possible responses to a prompt, while second-order uncertainty (uncertainty about uncertainty) quantifies indeterminacy in the underlying probability model itself. We introduce general-purpose prompting and post-processing procedures to directly elicit and quantify both orders of uncertainty, and demonstrate their effectiveness across diverse settings. Our approach enables more faithful uncertainty reporting from LLMs, improving credibility and supporting downstream decision-making.