Toward Template-Free Explainability for Monte Carlo Tree Search

TL;DR

This paper introduces a framework that uses large language models to generate natural language explanations for Monte Carlo Tree Search decisions, making the process more interpretable without relying on formal logic constraints.

Contribution

The authors propose an end-to-end LLM-based approach to explain MCTS decisions from search traces, eliminating the need for handcrafted formal representations.

Findings

LLMs can generate evidence-grounded explanations for MCTS decisions

The framework effectively maps natural questions to intent categories

Targeted expansion improves explanation quality

Abstract

Probabilistic search algorithms, such as Monte Carlo Tree Search (MCTS), have proven very effective in solving sequential decision-making tasks under uncertainty. However, interpreting asymmetric search trees that incorporate bandit-based tree traversal and simulation-based value estimation is difficult for end users based solely on raw tree statistics. While prior work requires hand-crafted formal logic constraints that must be updated when the problem changes, we present a framework that enables large language models (LLMs) to generate evidence-grounded explanations of MCTS decisions from recorded search traces in an end-to-end manner. Our framework maps natural-language questions to a structured set of intent categories, determines whether the existing tree contains sufficient evidence, triggers targeted expansion when needed, and generates explanations using tree statistics such as visit counts, value estimates, and risk information. Experimental results provide the first evidence that LLMs can serve as end-to-end explainers for probabilistic search, without requiring intermediate formal representations.