Investigating Scale Independent UCT Exploration Factor Strategies

TL;DR

This paper proposes adaptive strategies for setting the UCT exploration constant in tree search algorithms, making them robust to different reward scales across various games, and demonstrates their effectiveness through experiments.

Contribution

The paper introduces five new lambda-strategies for UCT exploration, including a data-driven method using Q-value standard deviation, improving performance across diverse tasks.

Findings

The proposed lambda = 2 * standard deviation method outperforms existing strategies.

Adaptive lambda strategies achieve better peak performance and robustness.

Experimental results span a wide range of game environments.

Abstract

The Upper Confidence Bounds For Trees (UCT) algorithm is not agnostic to the reward scale of the game it is applied to. For zero-sum games with the sparse rewards of $\{-1,0,1\}$ at the end of the game, this is not a problem, but many games often feature dense rewards with hand-picked reward scales, causing a node's Q-value to span different magnitudes across different games. In this paper, we evaluate various strategies for adaptively choosing the UCT exploration constant $\lambda$, called $\lambda$-strategies, that are agnostic to the game's reward scale. These $\lambda$-strategies include those proposed in the literature as well as five new strategies. Given our experimental results, we recommend using one of our newly suggested $\lambda$-strategies, which is to choose $\lambda$ as $2 \cdot \sigma$ where $\sigma$ is the empirical standard deviation of all state-action pairs' Q-values of the search tree. This method outperforms existing $\lambda$-strategies across a wide range of tasks both in terms of a single parameter value and the peak performances obtained by optimizing all available parameters.