When More is Less: Understanding Chain-of-Thought Length in LLMs

TL;DR

This paper reveals that longer Chain-of-Thought reasoning in LLMs does not always improve performance, showing an inverted U-shaped relationship, and introduces a theoretical model to optimize CoT length based on task difficulty and model capability.

Contribution

It provides a theoretical framework explaining CoT length effects, uncovers the simplicity bias in models, and offers practical methods for adaptive CoT calibration to enhance reasoning accuracy.

Findings

Performance follows an inverted U-shaped curve with CoT length.

Optimal CoT length increases with task difficulty but decreases with model capability.

Training with optimal CoT length improves reasoning accuracy.

Abstract

Large Language Models (LLMs) employ Chain-of-Thought (CoT) reasoning to deconstruct complex problems. While longer CoTs are often presumed superior, this paper challenges that notion, arguing that longer is not always better. Drawing on combined evidence from real-world observations, controlled experiments, and theoretical analysis, we demonstrate that task accuracy typically follows an inverted U-shaped curve with CoT length, where performance initially improves but eventually decreases as the number of CoT steps increases. With controlled experiments, we further uncover the scaling behaviors of the optimal CoT length: it increases with task difficulty but decreases with model capability, exposing an inherent simplicity bias where more capable models favor shorter, more efficient CoT reasoning. This bias is also evident in Reinforcement Learning (RL) training, where models gravitate towards shorter CoTs as their accuracy improves. To have a deep understanding of these dynamics, we establish a simple theoretical model that formally proves these phenomena, including the optimal length's scaling laws and the emergence of simplicity bias during RL. Guided by this framework, we demonstrate significant practical benefits from training with optimally-lengthed CoTs and employing length-aware filtering at inference. These findings offer both a principled understanding of the "overthinking" phenomenon and multiple practical guidelines for CoT calibration, enabling LLMs to achieve optimal reasoning performance with adaptive CoTs tailored to task complexity and model capability.