Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure

TL;DR

This paper models latent chain-of-thought reasoning as a causal process in representation space, analyzing how internal steps influence correctness and output trajectories across reasoning tasks.

Contribution

It introduces a causal modeling framework for latent reasoning steps, providing new insights into their necessity, influence propagation, and output commitment in latent chain-of-thought methods.

Findings

Latent steps act more like staged functionality than simple depth.

Early output bias differs from late representational commitment.

Causal analysis reveals non-local routing and influence patterns.

Abstract

Latent or continuous chain-of-thought methods replace explicit textual rationales with a number of internal latent steps, but these intermediate computations are difficult to evaluate beyond correlation-based probes. In this paper, we view latent chain-of-thought as a manipulable causal process in representation space by modeling latent steps as variables in a structural causal model (SCM) and analyzing their effects through step-wise $\mathrm{do}$-interventions. We study two representative paradigms (i.e., Coconut and CODI) on both mathematical and general reasoning tasks to investigate three key questions: (1) which steps are causally necessary for correctness and when answers become decidable early; (2) how does influence propagate across steps, and how does this structure compare to explicit CoT; and (3) do intermediate trajectories retain competing answer modes, and how does output-level commitment differ from representational commitment across steps. We find that latent-step budgets behave less like homogeneous extra depth and more like staged functionality with non-local routing, and we identify a persistent gap between early output bias and late representational commitment. These results motivate mode-conditional and stability-aware analyses -- and corresponding training/decoding objectives -- as more reliable tools for interpreting and improving latent reasoning systems. Code is available at https://github.com/J1mL1/causal-latent-cot.