PMCTS: Particle Monte Carlo Tree Search for Principled Parallelized Inference Time Scaling

TL;DR

PMCTS is a novel parallel Monte Carlo Tree Search algorithm that scales efficiently with parallel compute, maintains policy improvement guarantees, and outperforms heuristic baselines in various domains.

Contribution

Introduces Particle MCTS, the first principled parallel MCTS method suitable for neural network evaluations with formal guarantees.

Findings

PMCTS scales effectively with parallel compute.

PMCTS significantly outperforms heuristic-based baselines.

Maintains formal policy improvement guarantees.

Abstract

Monte Carlo Tree Search (MCTS) is a widely used approach for policy improvement through search with increasing popularity for real world applications. Due to the sequential and deterministic nature of its search, runtime-scaling of MCTS with parallel compute remains a major challenge. We introduce Particle MCTS (PMCTS), to our knowledge the first principled parallel MCTS algorithm which is suited for neural network evaluations and can preserve formal policy improvement guarantees. Empirically, PMCTS scales well with parallel compute and significantly outperforms the popular heuristic-based baselines across domains.