Can Aha Moments Be Fake? Identifying True and Decorative Thinking Steps in Chain-of-Thought
Jiachen Zhao, Yiyou Sun, Weiyan Shi, Dawn Song

TL;DR
This paper introduces the True Thinking Score to distinguish between genuinely causal reasoning steps and superficial ones in LLMs' chain-of-thought, revealing many steps are decorative rather than truly causal.
Contribution
It proposes a quantitative metric for assessing the causal contribution of reasoning steps in LLMs and demonstrates how models can be steered to follow or ignore specific reasoning steps.
Findings
Only a small percentage of reasoning steps causally influence the final prediction.
LLMs often verbalize reasoning steps without internalizing them.
Steering can direct models to focus on true reasoning steps.
Abstract
Large language models can generate long chain-of-thought (CoT) reasoning, but it remains unclear whether the verbalized steps reflect the models' internal thinking. In this work, we propose a True Thinking Score (TTS) to quantify the causal contribution of each step in CoT to the model's final prediction. Our experiments show that LLMs often interleave between true-thinking steps (which are genuinely used to compute the final output) and decorative-thinking steps (which give the appearance of reasoning but have minimal causal influence). We reveal that only a small subset of the total reasoning steps causally drive the model's prediction: e.g., on AIME, only an average of 2.3% of reasoning steps in CoT have a TTS >= 0.7 (range: 0-1) for Qwen-2.5. Furthermore, we find that LLMs can be steered to internally follow or disregard specific steps in their verbalized CoT using the identified…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
