Plan of Thoughts: Heuristic-Guided Problem Solving with Large Language Models

TL;DR

This paper introduces a planning-based method using POMDPs and heuristics for large language models to improve multi-step reasoning, achieving higher success rates and better performance on complex tasks.

Contribution

It formalizes a POMDP-based framework for multi-step problem solving with LMs and demonstrates its effectiveness with the POMCP solver on the Game of 24.

Findings

Achieved 89.4% success rate on the Game of 24.

Outperforms existing approaches in success rate and anytime performance.

Provides a formal planning framework for LMs in multi-step reasoning.

Abstract

While language models (LMs) offer significant capability in zero-shot reasoning tasks across a wide range of domains, they do not perform satisfactorily in problems which requires multi-step reasoning. Previous approaches to mitigate this involves breaking a larger, multi-step task into sub-tasks and asking the language model to generate proposals ("thoughts") for each sub-task and using exhaustive planning approaches such as DFS to compose a solution. In this work, we leverage this idea to introduce two new contributions: first, we formalize a planning-based approach to perform multi-step problem solving with LMs via Partially Observable Markov Decision Processes (POMDPs), with the LM's own reflections about the value of a state used as a search heuristic; second, leveraging the online POMDP solver POMCP, we demonstrate a superior success rate of 89.4% on the Game of 24 task as compared to existing approaches while also offering better anytime performance characteristics than fixed tree-search which is used previously. Taken together, these contributions allow modern LMs to decompose and solve larger-scale reasoning tasks more effectively.