Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

TL;DR

This paper investigates the effectiveness of Chain-of-Thought prompting in large language models, revealing its reliance on training data distribution and its limitations beyond in-distribution scenarios.

Contribution

It introduces a data distribution perspective and a controllable environment, DataAlchemy, to systematically analyze when CoT reasoning succeeds or fails.

Findings

CoT reasoning is effective within training data distribution.

Performance degrades significantly when test data diverges from training distribution.

CoT reasoning is a brittle phenomenon that does not generalize well beyond training conditions.

Abstract

Chain-of-Thought (CoT) prompting has been shown to be effective in eliciting structured reasoning (i.e., CoT reasoning) from large language models (LLMs). Regardless of its popularity, recent studies expose its failures in some reasoning tasks, raising fundamental questions about the nature of CoT reasoning. In this work, we propose a data distribution lens to understand when and why CoT reasoning succeeds or fails. We hypothesize that CoT reasoning reflects a structured inductive bias learned from in-distribution data, enabling models to conditionally generate reasoning trajectories that approximate those observed during training. As such, the effectiveness of CoT reasoning is fundamentally governed by the nature and degree of distribution discrepancy between training data and test queries. Guided by this lens, we dissect CoT reasoning via three dimensions: task, length, and format. To test the hypothesis, we introduce DataAlchemy, an abstract and fully controllable environment that trains LLMs from scratch and systematically probes them under various distribution conditions. Through rigorous controlled experiments, we reveal that CoT reasoning is a brittle mirage when it is pushed beyond training distributions, emphasizing the ongoing challenge of achieving genuine and generalizable reasoning.