RoT: Enhancing Large Language Models with Reflection on Search Trees

TL;DR

RoT enhances large language models by enabling them to reflect on and learn from previous search experiences, leading to improved reasoning and planning performance across various tree-search prompting methods.

Contribution

The paper introduces RoT, a novel framework that uses LLM-based reflection and a state selection method to improve reasoning in tree-search prompting methods.

Findings

RoT significantly improves LLM performance in reasoning tasks.

RoT benefits both tree-search and non-tree-search prompting methods.

Guidelines generated by RoT help prevent repeated mistakes.

Abstract

Large language models (LLMs) have demonstrated impressive capability in reasoning and planning when integrated with tree-search-based prompting methods. However, since these methods ignore the previous search experiences, they often make the same mistakes in the search process. To address this issue, we introduce Reflection on search Trees (RoT), an LLM reflection framework designed to improve the performance of tree-search-based prompting methods. It uses a strong LLM to summarize guidelines from previous tree search experiences to enhance the ability of a weak LLM. The guidelines are instructions about solving this task through tree search which can prevent the weak LLMs from making similar mistakes in the past search process. In addition, we proposed a novel state selection method, which identifies the critical information from historical search processes to help RoT generate more specific and meaningful guidelines. In our extensive experiments, we find that RoT significantly improves the performance of LLMs in reasoning or planning tasks with various tree-search-based prompting methods (e.g., BFS and MCTS). Non-tree-search-based prompting methods such as Chain-of-Thought (CoT) can also benefit from RoT guidelines since RoT can provide task-specific knowledge collected from the search experience.