Accelerating Monte Carlo Tree Search with Probability Tree State Abstraction

TL;DR

This paper introduces a probability tree state abstraction method to enhance Monte Carlo Tree Search efficiency, reducing search space and accelerating training in state-of-the-art algorithms like MuZero.

Contribution

The paper proposes a novel probability tree state abstraction with theoretical guarantees, improving MCTS efficiency and reducing search space in complex tasks.

Findings

Achieved 10%-45% reduction in search space.

Enhanced training speed of MuZero-based algorithms.

Provided theoretical bounds for abstraction errors.

Abstract

Monte Carlo Tree Search (MCTS) algorithms such as AlphaGo and MuZero have achieved superhuman performance in many challenging tasks. However, the computational complexity of MCTS-based algorithms is influenced by the size of the search space. To address this issue, we propose a novel probability tree state abstraction (PTSA) algorithm to improve the search efficiency of MCTS. A general tree state abstraction with path transitivity is defined. In addition, the probability tree state abstraction is proposed for fewer mistakes during the aggregation step. Furthermore, the theoretical guarantees of the transitivity and aggregation error bound are justified. To evaluate the effectiveness of the PTSA algorithm, we integrate it with state-of-the-art MCTS-based algorithms, such as Sampled MuZero and Gumbel MuZero. Experimental results on different tasks demonstrate that our method can accelerate the training process of state-of-the-art algorithms with 10%-45% search space reduction.