Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers

TL;DR

This paper investigates the fundamental limits of chain-of-thought reasoning in hard-attention transformers, providing tight lower bounds on the number of reasoning steps needed for various algorithmic problems, thus clarifying their computational power.

Contribution

It establishes systematic lower bounds on chain-of-thought steps in hard-attention transformers for multiple problems, advancing understanding of their capabilities and limitations.

Findings

Lower bounds are tight up to logarithmic factors for several problems.

Transformers require a significant number of reasoning steps for certain tasks.

Results challenge assumptions about the efficiency of chain-of-thought reasoning.

Abstract

Chain-of-thought reasoning and scratchpads have emerged as critical tools for enhancing the computational capabilities of transformers. While theoretical results show that polynomial-length scratchpads can extend transformers' expressivity from $TC^0$ to $PTIME$, their required length remains poorly understood. Empirical evidence even suggests that transformers need scratchpads even for many problems in $TC^0$, such as Parity or Multiplication, challenging optimistic bounds derived from circuit complexity. In this work, we initiate the study of systematic lower bounds for the number of chain-of-thought steps across different algorithmic problems, in the hard-attention regime. We study a variety of algorithmic problems, and provide bounds that are tight up to logarithmic factors. Overall, these results contribute to emerging understanding of the power and limitations of chain-of-thought reasoning.