Certain but not Probable? Differentiating Certainty from Probability in LLM Token Outputs for Probabilistic Scenarios

TL;DR

This paper examines whether large language models' token certainty measures accurately reflect theoretical probabilities in probabilistic scenarios, revealing divergence despite correct responses.

Contribution

It demonstrates that token-level certainty metrics do not reliably indicate true probabilistic alignment in LLM outputs, highlighting limitations in current uncertainty quantification methods.

Findings

Models achieve perfect response accuracy in probabilistic tasks.

Token probabilities and entropy diverge from theoretical distributions.

Certainty measures may not reflect true probabilistic alignment.

Abstract

Reliable uncertainty quantification (UQ) is essential for ensuring trustworthy downstream use of large language models, especially when they are deployed in decision-support and other knowledge-intensive applications. Model certainty can be estimated from token logits, with derived probability and entropy values offering insight into performance on the prompt task. However, this approach may be inadequate for probabilistic scenarios, where the probabilities of token outputs are expected to align with the theoretical probabilities of the possible outcomes. We investigate the relationship between token certainty and alignment with theoretical probability distributions in well-defined probabilistic scenarios. Using GPT-4.1 and DeepSeek-Chat, we evaluate model responses to ten prompts involving probability (e.g., roll a six-sided die), both with and without explicit probability cues in the prompt (e.g., roll a fair six-sided die). We measure two dimensions: (1) response validity with respect to scenario constraints, and (2) alignment between token-level output probabilities and theoretical probabilities. Our results indicate that, while both models achieve perfect in-domain response accuracy across all prompt scenarios, their token-level probability and entropy values consistently diverge from the corresponding theoretical distributions.