Empirical-MCTS: Continuous Agent Evolution via Dual-Experience Monte Carlo Tree Search

TL;DR

Empirical-MCTS introduces a dual-loop framework that enhances reasoning in Large Language Models by combining local exploration with global memory optimization, enabling continuous learning from experience during inference.

Contribution

The paper presents Empirical-MCTS, a novel framework that transforms stateless search into a continuous learning process using pairwise feedback and global memory management.

Findings

Outperforms traditional stateless MCTS on complex reasoning benchmarks.

Demonstrates significant improvements in reasoning accuracy and efficiency.

Validates the importance of empirical experience accumulation in reasoning tasks.

Abstract

Inference-time scaling strategies, particularly Monte Carlo Tree Search (MCTS), have significantly enhanced the reasoning capabilities of Large Language Models (LLMs). However, current approaches remain predominantly stateless, discarding successful reasoning patterns after each problem instance and failing to mimic the empirical accumulation of wisdom characteristic of human problem-solving. To bridge this gap, we introduce Empirical-MCTS, a dual-loop framework that transforms stateless search into a continuous, non-parametric learning process. The framework unifies local exploration with global memory optimization through two novel mechanisms: Pairwise-Experience-Evolutionary Meta-Prompting (PE-EMP) and a Memory Optimization Agent. PE-EMP functions as a reflexive optimizer within the local search, utilizing pairwise feedback to dynamically synthesize adaptive criteria and evolve meta-prompts (system prompts) in real-time. Simultaneously, the Memory Optimization Agent manages a global repository as a dynamic policy prior, employing atomic operations to distill high-quality insights across problems. Extensive evaluations on complex reasoning benchmarks, including AIME25, ARC-AGI-2, and MathArena Apex, demonstrate that Empirical-MCTS significantly outperforms both stateless MCTS strategies and standalone experience-driven agents. These results underscore the critical necessity of coupling structured search with empirical accumulation for mastering complex, open-ended reasoning tasks.