Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces

TL;DR

This paper investigates the overcompleteness of language model reasoning traces, defining minimal cores that preserve predictions and revealing their geometric and theoretical properties across diverse benchmarks.

Contribution

It introduces metrics for analyzing reasoning trace overcompleteness, defines minimal cores, and provides empirical and theoretical insights into their properties and benefits.

Findings

46% of reasoning steps are removable while preserving answers in 86% of cases.

Top three steps account for 65% of necessity, indicating concentrated support.

Minimal cores improve trace separation, reduce intrinsic dimensionality, and transfer across models.

Abstract

Language models often generate long chain-of-thought traces, but it remains unclear how much of this reasoning is necessary for preserving the final prediction. We study this through the lens of overcomplete reasoning traces: generated traces that contain more intermediate steps than are needed to support the model's answer. We define the minimal core as the smallest subset of steps that preserves either the final answer or predictive distribution, and introduce metrics for compression ratio, redundancy mass, step necessity, and necessity concentration. Across six deliberative reasoning benchmarks spanning arithmetic, competition mathematics, expert scientific reasoning, and commonsense multi-hop QA, we find substantial overcompleteness: on average, 46% of steps are removable under greedy minimal-core extraction while preserving the original answer in 86% of cases. We also find that predictive support is concentrated: the top three steps account for 65% of measured necessity mass on average. Beyond compression, minimal cores expose a cleaner geometry of reasoning: compared with full traces, they improve correct-incorrect trace separation by 11 points, reduce estimated intrinsic dimensionality by 34%, and transfer across model families with 85% off-diagonal answer retention. Theoretically, we establish existence of minimal sufficient subsets, local irreducibility guarantees for greedy elimination, and certificates of overcompleteness and sparse necessity. Together, these results suggest that full reasoning traces are often verbose and overcomplete, while minimal cores isolate the effective support underlying language-model predictions.