State Machine of Thoughts: Leveraging Past Reasoning Trajectories for Enhancing Problem Solving

TL;DR

This paper introduces a state machine approach to record and utilize past reasoning trajectories in large language model agents, significantly improving their problem-solving efficiency in complex tasks.

Contribution

It proposes the State Machine of Thoughts (SMoT) that leverages past successful and failed reasoning paths to enhance future problem-solving performance.

Findings

SMoT improves problem-solving accuracy in the 24-point game.

SMoT enhances navigation efficiency in a taxi RL environment.

The approach effectively reuses past reasoning trajectories for better decision-making.

Abstract

Current Large Language Model-based agents reason within an exploration-evaluation framework, navigating problem-solving processes in a tree-like manner. However, these methods often neglect successful reasoning trajectories once a problem is resolved, leading to inefficient use of these trajectories for future analogous problems. To address this inefficiency, we adopt a state machine to record experience derived from previous reasoning trajectories. Within the state machine, states represent decomposed sub-problems, while state transitions reflect the dependencies among sub-problems. The state machine records both successful and failed trajectories. Utilizing the experience from the state machine, our proposed State Machine of Thoughts (SMoT) selects the most optimal sub-solutions and avoids incorrect ones. Our experiments show that SMoT can significantly improve problem-solving abilities in two exploration-intensive problems: the 24-point game and a taxi navigation reinforcement learning game.