SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition

TL;DR

SELT introduces a self-evaluation tree search framework for LLMs that enhances reasoning accuracy and robustness in complex tasks by leveraging a modified Monte Carlo Tree Search without external reward models.

Contribution

The paper presents a novel MCTS-based framework that improves LLM reasoning through intrinsic self-evaluation and task decomposition, without requiring task-specific fine-tuning.

Findings

Significant accuracy improvements on MMLU and Seal-Tools benchmarks.

Enhanced reasoning robustness and reduced hallucination in LLM outputs.

Operates effectively without external reward models or fine-tuning.

Abstract

While Large Language Models (LLMs) have achieved remarkable success in a wide range of applications, their performance often degrades in complex reasoning tasks. In this work, we introduce SELT (Self-Evaluation LLM Tree Search), a novel framework that leverages a modified Monte Carlo Tree Search (MCTS) to enhance LLM reasoning without relying on external reward models. By redefining the Upper Confidence Bound scoring to align with intrinsic self-evaluation capabilities of LLMs and decomposing the inference process into atomic subtasks augmented with semantic clustering at each node, SELT effectively balances exploration and exploitation, reduces redundant reasoning paths, and mitigates hallucination. We validate our approach on challenging benchmarks, including the knowledge-based MMLU and the Tool Learning dataset Seal-Tools, where SELT achieves significant improvements in answer accuracy and reasoning robustness compared to baseline methods. Notably, our framework operates without task-specific fine-tuning, demonstrating strong generalizability across diverse reasoning tasks. Relevant results and code are available at https://github.com/fairyshine/SELT .