Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Tasks

TL;DR

This paper investigates the internal mechanisms of Latent-CoT models, revealing how they perform step-by-step reasoning and under what conditions they produce faithful or shortcut solutions, using a detailed mechanistic analysis.

Contribution

It provides a mechanistic study of Latent-CoT models, identifying how they form intermediate states and when they rely on full versus partial latent reasoning paths.

Findings

CODI forms decodable bridge states for short hops

Final predictions often use late fusion of intermediates

Partial latent reasoning dominates in longer, harder tasks

Abstract

Latent Chain-of-Thought (Latent-CoT) aims to enable step-by-step computation without emitting long rationales, yet its mechanisms remain unclear. We study CODI, a continuous-thought teacher-student distillation model, on strictly sequential polynomial-iteration tasks. Using logit-lens decoding, linear probes, attention analysis, and activation patching, we localize intermediate-state representations and trace their routing to the final readout. On two- and three-hop tasks, CODI forms the full set of bridge states that become decodable across latent-thought positions, while the final input follows a separate near-direct route; predictions arise via late fusion at the end-of-thought boundary. For longer hop lengths, CODI does not reliably execute a full latent rollout, instead exhibiting a partial latent reasoning path that concentrates on late intermediates and fuses them with the last input at the answer readout position. Ablations show that this partial pathway can collapse under regime shifts, including harder optimization. Overall, we delineate when CODI-style latent-CoT yields faithful iterative computation versus compressed or shortcut strategies, and highlight challenges in designing robust latent-CoT objectives for sequential reasoning.