Policy Guided Tree Search for Enhanced LLM Reasoning

TL;DR

This paper introduces Policy-Guided Tree Search (PGTS), a reinforcement learning framework that improves large language model reasoning by dynamically guiding structured exploration, outperforming existing heuristic-based methods in efficiency and accuracy.

Contribution

The paper presents a novel reinforcement learning approach that learns policies for structured tree exploration, reducing reliance on heuristics and enhancing reasoning capabilities of LLMs.

Findings

PGTS outperforms existing methods on reasoning benchmarks.

It significantly reduces computational costs.

Demonstrates scalability across diverse reasoning tasks.

Abstract

Despite their remarkable capabilities, large language models often struggle with tasks requiring complex reasoning and planning. While existing approaches like Chain-of-Thought prompting and tree search techniques show promise, they are limited by their reliance on predefined heuristics and computationally expensive exploration strategies. We propose Policy-Guided Tree Search (PGTS), a framework that combines reinforcement learning with structured tree exploration to efficiently navigate reasoning paths. Our key innovation is a learned policy that dynamically decides between expanding, branching, backtracking, or terminating exploration, eliminating the need for manual heuristics or exhaustive search. Experiments across mathematical reasoning, logical deduction, and planning benchmarks demonstrate that PGTS achieves superior reasoning performance while significantly reducing computational costs compared to existing methods. These results establish PGTS as a scalable and effective solution for tackling complex reasoning tasks with LLMs.