How does Chain of Thought decompose complex tasks?

TL;DR

The paper analyzes how decomposing complex tasks into chain-of-thought sequences reduces classification error in language models, revealing a critical threshold for optimal decomposition depth.

Contribution

It introduces a power-law model of classification error scaling and identifies an optimal chain depth for task decomposition in language models.

Findings

Classification error scales as a power law with number of classes.

Decomposing tasks into smaller classification problems reduces error.

An optimal chain depth exists beyond which additional thinking does not improve performance.

Abstract

Many language tasks can be modeled as classification problems where a large language model (LLM) is given a prompt and selects one among many possible answers. We show that the classification error in such problems scales as a power law in the number of classes. This has a dramatic consequence: the prediction error can be reduced substantially by splitting the overall task into a sequence of smaller classification problems, each with the same number of classes ("degree"). This tree-structured decomposition models chain-of-thought (CoT). It has been observed that CoT-based predictors perform better when they "think", i.e., when they develop a deeper tree, thus decomposing the problem into a larger number of steps. We identify a critical threshold for the degree, below which thinking is detrimental, and above which there exists an optimal depth that minimizes the error. It is impossible to surpass this minimal error by increasing the depth of thinking.