An Efficient Algorithm for Thresholding Monte Carlo Tree Search

TL;DR

This paper presents a new efficient algorithm for the Thresholding Monte Carlo Tree Search problem, improving sample complexity and computational cost through a novel ratio-based modification of existing strategies.

Contribution

The paper introduces a $ ext{delta}$-correct sequential sampling algorithm with asymptotic optimality and a modified D-Tracking strategy that enhances empirical performance and reduces computational complexity.

Findings

Asymptotically optimal sample complexity achieved.

Significant empirical improvements over previous methods.

Reduced per-round computational cost from linear to logarithmic.

Abstract

We introduce the Thresholding Monte Carlo Tree Search problem, in which, given a tree $\mathcal{T}$ and a threshold $\theta$, a player must answer whether the root node value of $\mathcal{T}$ is at least $\theta$ or not. In the given tree, `MAX' or `MIN' is labeled on each internal node, and the value of a `MAX'-labeled (`MIN'-labeled) internal node is the maximum (minimum) of its child values. The value of a leaf node is the mean reward of an unknown distribution, from which the player can sample rewards. For this problem, we develop a $\delta$-correct sequential sampling algorithm based on the Track-and-Stop strategy that has asymptotically optimal sample complexity. We show that a ratio-based modification of the D-Tracking arm-pulling strategy leads to a substantial improvement in empirical sample complexity, as well as reducing the per-round computational cost from linear to logarithmic in the number of arms.