When does Chain-of-Thought Help: A Markovian Perspective

TL;DR

This paper models chain-of-thought prompting as a Markov chain to analyze when it improves reasoning, showing that shared transition kernels across steps are key for its effectiveness and quantifying the impact of noise.

Contribution

It introduces a Markovian framework to explain CoT's effectiveness, identifying transition alignment as crucial and providing theoretical and empirical validation.

Findings

Shared transition kernels enhance CoT effectiveness.

Differences in transitions can negate CoT benefits.

Noise in intermediate steps affects CoT gains.

Abstract

Chain-of-Thought (CoT) prompting is a widely used inference-time technique for improving reasoning, yet its gains are uneven across tasks. We analyze when and why CoT helps by modeling the step-wise reasoning trajectory as a Markov chain. Each intermediate step is a state and the dependence between steps is captured by a transition kernel. Our theory identifies transition alignment, whether instances share a common step-wise transition kernel, as the key determinant of CoT's effectiveness. When transitions are identical across steps, CoT reduces inference-time sample complexity: fewer context sample trajectories suffice to recover the final decision. In contrast, when transitions differ across steps, these gains can vanish. We further quantify how noise in intermediate steps modulates CoT's benefit. Beyond theory, we design synthetic benchmarks that isolate these factors to complement prior results on real-world tasks and to empirically validate our predictions.