Can Aha Moments Be Fake? Identifying True and Decorative Thinking Steps in Chain-of-Thought

TL;DR

This paper introduces the True Thinking Score to distinguish between genuinely causal reasoning steps and superficial ones in LLMs' chain-of-thought, revealing many steps are decorative rather than truly causal.

Contribution

It proposes a quantitative metric for assessing the causal contribution of reasoning steps in LLMs and demonstrates how models can be steered to follow or ignore specific reasoning steps.

Findings

Only a small percentage of reasoning steps causally influence the final prediction.

LLMs often verbalize reasoning steps without internalizing them.

Steering can direct models to focus on true reasoning steps.

Abstract

Large language models can generate long chain-of-thought (CoT) reasoning, but it remains unclear whether the verbalized steps reflect the models' internal thinking. In this work, we propose a True Thinking Score (TTS) to quantify the causal contribution of each step in CoT to the model's final prediction. Our experiments show that LLMs often interleave between true-thinking steps (which are genuinely used to compute the final output) and decorative-thinking steps (which give the appearance of reasoning but have minimal causal influence). We reveal that only a small subset of the total reasoning steps causally drive the model's prediction: e.g., on AIME, only an average of 2.3% of reasoning steps in CoT have a TTS >= 0.7 (range: 0-1) for Qwen-2.5. Furthermore, we find that LLMs can be steered to internally follow or disregard specific steps in their verbalized CoT using the identified TrueThinking direction. We highlight that self-verification steps in CoT (i.e., aha moments) can be decorative, while steering along the TrueThinking direction can force internal reasoning over these steps. Overall, our work reveals that LLMs often verbalize reasoning steps without performing them internally, challenging the efficiency of LLM reasoning and the trustworthiness of CoT.