When Chain-of-Thought Fails, the Solution Hides in the Hidden States

TL;DR

This paper investigates how chain-of-thought reasoning in language models contains recoverable, token-level information that can be used to improve answer accuracy through activation patching, revealing insights into reasoning representation.

Contribution

It provides a mechanistic causal analysis showing that task-relevant information is stored in hidden states, especially in mid-to-late layers, and can be leveraged to enhance reasoning accuracy.

Findings

Patching hidden states improves answer accuracy over original CoT and direct prompting.

Task-relevant information is concentrated in mid-to-late layers and appears earlier in correct traces.

Shorter patched outputs can outperform full CoT traces in accuracy.

Abstract

Whether intermediate reasoning is computationally useful or merely explanatory depends on whether chain-of-thought (CoT) tokens contain task-relevant information. We present a mechanistic causal analysis of CoT on GSM8K using activation patching: transferring token-level hidden states from a CoT generation to a direct-answer run for the same question, then measuring the effect on final-answer accuracy. Across models, generating after patching yields substantially higher accuracy than both direct-answer prompting and the original CoT trace, revealing that individual CoT tokens can encode sufficient information to recover the correct answer, even when the original trace is incorrect. This task-relevant information is more prevalent in correct than incorrect CoT runs and is unevenly distributed across tokens, concentrating in mid-to-late layers and appearing earlier in the reasoning trace. Moreover, patching language tokens such as verbs and entities carry task-solving information that steers generation toward correct reasoning, whereas mathematical tokens encode answer-proximal content that rarely succeeds. Patched outputs are often shorter and yet exceed the accuracy of a full CoT trace, suggesting complete reasoning chains are not always necessary. Together, these findings demonstrate that CoT encodes recoverable, token-level problem-solving information, offering new insight into how reasoning is represented and where it breaks down.