A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search

TL;DR

This paper introduces a unified mathematical framework for enhancing exploration and convergence in Monte-Carlo Tree Search by using a new backup operator, entropy regularization, and the $eta$-divergence, validated across various decision-making problems.

Contribution

It proposes a novel theoretical framework based on $eta$-divergence that unifies existing MCTS methods and introduces new techniques for improved exploration and convergence.

Findings

The proposed methods improve convergence rates in MCTS.

The framework effectively balances exploration and exploitation.

Empirical results show superior performance on Atari games and POMDPs.

Abstract

Monte-Carlo Tree Search (MCTS) is a class of methods for solving complex decision-making problems through the synergy of Monte-Carlo planning and Reinforcement Learning (RL). The highly combinatorial nature of the problems commonly addressed by MCTS requires the use of efficient exploration strategies for navigating the planning tree and quickly convergent value backup methods. These crucial problems are particularly evident in recent advances that combine MCTS with deep neural networks for function approximation. In this work, we propose two methods for improving the convergence rate and exploration based on a newly introduced backup operator and entropy regularization. We provide strong theoretical guarantees to bound convergence rate, approximation error, and regret of our methods. Moreover, we introduce a mathematical framework based on the use of the $\alpha$-divergence for backup and exploration in MCTS. We show that this theoretical formulation unifies different approaches, including our newly introduced ones, under the same mathematical framework, allowing to obtain different methods by simply changing the value of $\alpha$. In practice, our unified perspective offers a flexible way to balance between exploration and exploitation by tuning the single $\alpha$ parameter according to the problem at hand. We validate our methods through a rigorous empirical study from basic toy problems to the complex Atari games, and including both MDP and POMDP problems.