Single-Agent Optimization Through Policy Iteration Using Monte-Carlo Tree Search

TL;DR

This paper introduces a novel Monte-Carlo Tree Search variant with normalization, parallelization, and a learned policy network, improving single-agent optimization performance in the game SameGame.

Contribution

It presents a new MCTS-based algorithm with normalization, virtual loss for parallelization, and a self-trained policy network for single-agent optimization.

Findings

Outperforms baseline algorithms on various board sizes

Competitive with state-of-the-art search methods on benchmark positions

Effective in optimizing single-agent game scenarios

Abstract

The combination of Monte-Carlo Tree Search (MCTS) and deep reinforcement learning is state-of-the-art in two-player perfect-information games. In this paper, we describe a search algorithm that uses a variant of MCTS which we enhanced by 1) a novel action value normalization mechanism for games with potentially unbounded rewards (which is the case in many optimization problems), 2) defining a virtual loss function that enables effective search parallelization, and 3) a policy network, trained by generations of self-play, to guide the search. We gauge the effectiveness of our method in "SameGame"---a popular single-player test domain. Our experimental results indicate that our method outperforms baseline algorithms on several board sizes. Additionally, it is competitive with state-of-the-art search algorithms on a public set of positions.