A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search
Tuan Dam, Carlo D'Eramo, Jan Peters, Joni Pajarinen

TL;DR
This paper introduces a unified mathematical framework for enhancing exploration and convergence in Monte-Carlo Tree Search by using a new backup operator, entropy regularization, and the $eta$-divergence, validated across various decision-making problems.
Contribution
It proposes a novel theoretical framework based on $eta$-divergence that unifies existing MCTS methods and introduces new techniques for improved exploration and convergence.
Findings
The proposed methods improve convergence rates in MCTS.
The framework effectively balances exploration and exploitation.
Empirical results show superior performance on Atari games and POMDPs.
Abstract
Monte-Carlo Tree Search (MCTS) is a class of methods for solving complex decision-making problems through the synergy of Monte-Carlo planning and Reinforcement Learning (RL). The highly combinatorial nature of the problems commonly addressed by MCTS requires the use of efficient exploration strategies for navigating the planning tree and quickly convergent value backup methods. These crucial problems are particularly evident in recent advances that combine MCTS with deep neural networks for function approximation. In this work, we propose two methods for improving the convergence rate and exploration based on a newly introduced backup operator and entropy regularization. We provide strong theoretical guarantees to bound convergence rate, approximation error, and regret of our methods. Moreover, we introduce a mathematical framework based on the use of the -divergence for backup…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsArtificial Intelligence in Games · Reinforcement Learning in Robotics · Sports Analytics and Performance
