Reasoning about Reasoning: BAPO Bounds on Chain-of-Thought Token Complexity in LLMs

TL;DR

This paper establishes fundamental lower bounds on the number of reasoning tokens needed for chain-of-thought in large language models, revealing linear scaling requirements and bottlenecks in inference-time compute.

Contribution

It extends the BAPO model to prove linear lower bounds on reasoning tokens for key tasks, providing theoretical insights into CoT complexity and empirical validation.

Findings

Reasoning tokens scale linearly with input size in key tasks.

Models fail when reasoning budgets are too small, aligning with theoretical bounds.

The study offers a framework to analyze optimal reasoning length in LLMs.

Abstract

Inference-time scaling via chain-of-thought (CoT) reasoning is a major driver of state-of-the-art LLM performance, but it comes with substantial latency and compute costs. We address a fundamental theoretical question: how many reasoning tokens are required to solve a problem as input size grows? By extending the bounded attention prefix oracle (BAPO) model--an abstraction of LLMs that quantifies the information flow required to solve a task--we prove lower bounds on the CoT tokens required for three canonical BAPO-hard tasks: binary majority, triplet matching, and graph reachability. We show that each requires $\Omega(n)$ reasoning tokens when the input size is $n$. We complement these results with matching or near-matching upper bounds via explicit constructions. Finally, our experiments with frontier reasoning models show approximately linear reasoning token scaling on these tasks and failures when constrained to smaller reasoning budgets, consistent with our theoretical lower bounds. Together, our results identify fundamental bottlenecks in inference-time compute through CoT and offer a principled tool for analyzing optimal reasoning length.