Caterpillar of Thoughts: The Optimal Test-Time Algorithm for Large Language Models

TL;DR

This paper models test-time computation for large language models as a Markov chain interaction, characterizes the optimal algorithm as a caterpillar tree, and introduces CaT, a new method that improves success rate and efficiency.

Contribution

It provides a theoretical framework for optimal test-time algorithms, characterizes the structure as a caterpillar tree, and proposes CaT, a novel algorithm that outperforms existing methods.

Findings

CaT achieves higher success rates than ToT.

CaT reduces the number of token generations.

The optimal test-time algorithm forms a caterpillar tree structure.

Abstract

Large language models (LLMs) can often produce substantially better outputs when allowed to use additional test-time computation, such as sampling, chain of thought, backtracking, or revising partial solutions. Despite the growing empirical success of such techniques, there is limited theoretical understanding of how inference time computation should be structured, or what constitutes an optimal use of a fixed computation budget. We model test-time computation as an algorithm interacting with a Markov chain: at any point, the algorithm may resume generation from any previously observed state. That is, unlike standard Markov chains where the states are drawn passively, we allow the algorithm to backtrack to any previously observed state of the Markov chain at any time. Many of the existing test-time algorithms, such as Chain-of-Thought (CoT) (Wei et al., 2023), Tree-of-Thoughts (ToT) (Yao et al., 2023), or Best-of-$k$ (Brown et al., 2024) could be seen as specific algorithms in this model. We prove that while backtracking can reduce the number of generations exponentially, a very limited form of backtracking is theoretically sufficient. Namely, we show that the optimal algorithm always generates a caterpillar tree. That is, if we remove the leaves of the state tree generated by the optimal algorithm, we obtain a path. Motivated by our characterization of the optimal algorithm, we present Caterpillar of Thoughts (CaT), a new test-time computation algorithm, reducing the number of token/state generations. Our empirical evaluation shows that CaT, compared to ToT, achieves a better success rate while also reducing the number of token generations.