The Stepwise Informativeness Assumption: Why are Entropy Dynamics and Reasoning Correlated in LLMs?

TL;DR

This paper introduces the Stepwise Informativeness Assumption (SIA), explaining why entropy dynamics in LLMs correlate with correctness, supported by theoretical derivation and empirical validation across multiple models and benchmarks.

Contribution

It formalizes SIA as a principle explaining entropy-correctness correlation, deriving observable signatures, and empirically validating it across diverse models and reasoning tasks.

Findings

SIA naturally emerges from maximum-likelihood training on reasoning traces.

Correct reasoning traces show characteristic entropy patterns.

Training induces the SIA in large language models.

Abstract

Recent work uses entropy-based signals at multiple representation levels to study reasoning in large language models, but the field remains largely empirical. A central unresolved puzzle is why internal entropy dynamics, defined under the predictive distribution of a model, correlate so robustly with external correctness given by the ground-truth answer. In this paper, we argue that this correlation arises because autoregressive models reason correctly when they accumulate information about the true answer via answer-informative prefixes. We formalize this intuition via the Stepwise Informativeness Assumption (SIA), which states that reasoning prefixes accumulate answer-relevant information in expectation as generation progresses. We show that SIA naturally emerges from maximum-likelihood optimization on human reasoning traces and is reinforced by standard fine-tuning and reinforcement-learning pipelines. We then derive observable signatures of SIA linking conditional answer entropy dynamics to correctness. We empirically test SIA across multiple reasoning benchmarks (GSM8K, ARC, SVAMP) and a diverse set of open-weight LLMs (Gemma-2, LLaMA-3.2, Qwen-2.5, DeepSeek and Olmo variants), showing that training induces it and that correct traces exhibit characteristic conditional answer entropy patterns.