Super-Exponential Regret for UCT, AlphaGo and Variants

TL;DR

This paper refines lower bound proofs for UCT and its variants, including AlphaGo's MCTS, demonstrating they can incur super-exponential regret in certain environments, correcting previous oversights in the proofs.

Contribution

It provides corrected and extended lower bound proofs for UCT and AlphaGo's MCTS, showing they can suffer super-exponential regret in specific scenarios.

Findings

UCT can have super-exponential regret on D-chain environments.

Polynomial UCT variants also exhibit super-exponential regret.

The proofs for these bounds are corrected and extended to AlphaGo's MCTS.

Abstract

We improve the proofs of the lower bounds of Coquelin and Munos (2007) that demonstrate that UCT can have $\exp(\dots\exp(1)\dots)$ regret (with $\Omega(D)$ exp terms) on the $D$-chain environment, and that a `polynomial' UCT variant has $\exp_2(\exp_2(D - O(\log D)))$ regret on the same environment -- the original proofs contain an oversight for rewards bounded in $[0, 1]$, which we fix in the present draft. We also adapt the proofs to AlphaGo's MCTS and its descendants (e.g., AlphaZero, Leela Zero) to also show $\exp_2(\exp_2(D - O(\log D)))$ regret.