The Detection-Extraction Gap: Models Know the Answer Before They Can Say It

TL;DR

This paper uncovers the detection-extraction gap in reasoning models, showing they often generate beyond the answer point and proposing a method to improve extraction efficiency and accuracy using free continuations.

Contribution

It formalizes the detection-extraction gap phenomenon and introduces BAEE, a black-box early exit method that reduces generation steps while enhancing answer extraction accuracy.

Findings

52--88% of chain-of-thought tokens are produced after the answer is recoverable.

BAEE reduces serial generation by 70--78% and improves accuracy by 1--5 percentage points.

Early exit prevents overwriting in thinking-mode models, gaining up to 5.8pp.

Abstract

Modern reasoning models continue generating long after the answer is already determined. Across five model configurations, two families, and three benchmarks, we find that 52--88% of chain-of-thought tokens are produced after the answer is recoverable from a partial prefix. This post-commitment generation reveals a structural phenomenon: the detection-extraction gap. Free continuations from early prefixes recover the correct answer even at 10% of the trace, while forced extraction fails on 42% of these cases. The answer is recoverable from the model state, yet prompt-conditioned decoding fails to extract it. We formalize this mismatch via a total-variation bound between free and forced continuation distributions, yielding quantitative estimates of suffix-induced shift. Exploiting this asymmetry, we propose Black-box Adaptive Early Exit (BAEE), which uses free continuations for both detection and extraction, truncating 70--78% of serial generation while improving accuracy by 1--5pp across all models. For thinking-mode models, early exit prevents post-commitment overwriting, yielding gains of up to 5.8pp; a cost-optimized variant achieves 68--73% reduction at a median of 9 API calls. Code is available at https://github.com/EdWangLoDaSc/know2say.