Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models

TL;DR

Boosting of Thoughts (BoT) is an automated iterative prompting framework that enhances large language models' problem-solving abilities by exploring, self-evaluating, and revising reasoning trees through trial-and-error to improve accuracy on complex problems.

Contribution

Introduces BoT, a novel automated prompting method that iteratively explores and refines reasoning strategies without initial examples, improving problem-solving performance.

Findings

BoT outperforms existing prompting methods on complex mathematical problems.

BoT achieves higher problem-solving accuracy with GPT-4 and Llama2.

Iterative self-evaluation enhances reasoning quality in LLMs.

Abstract

The reasoning performance of Large Language Models (LLMs) on a wide range of problems critically relies on chain-of-thought prompting, which involves providing a few chain of thought demonstrations as exemplars in prompts. Recent work, e.g., Tree of Thoughts, has pointed out the importance of exploration and self-evaluation in reasoning step selection for complex problem solving. In this paper, we present Boosting of Thoughts (BoT), an automated prompting framework for problem solving with LLMs by iteratively exploring and self-evaluating many trees of thoughts in order to acquire an ensemble of trial-and-error reasoning experiences, which will serve as a new form of prompting to solve the complex problem. Starting from a simple prompt without requiring examples, BoT iteratively explores and evaluates a large collection of reasoning steps, and more importantly, uses error analysis obtained from the LLM on them to explicitly revise prompting, which in turn enhances reasoning step generation, until a final answer is attained. Our experiments with GPT-4 and Llama2 across extensive complex mathematical problems demonstrate that BoT consistently achieves higher or comparable problem-solving rates than other advanced prompting approaches.