Mastering the Game of Go with Self-play Experience Replay

TL;DR

This paper introduces QZero, a model-free reinforcement learning algorithm that learns to play Go at a high level through self-play and experience replay, without relying on search or human data.

Contribution

QZero is the first to demonstrate that model-free RL can master Go, using off-policy learning and a single Q-network, achieving performance comparable to AlphaGo.

Findings

QZero matches AlphaGo's performance after 5 months of training.

It demonstrates the viability of model-free RL for complex strategy games.

Off-policy learning effectively trains large-scale Go agents.

Abstract

The game of Go has long served as a benchmark for artificial intelligence, demanding sophisticated strategic reasoning and long-term planning. Previous approaches such as AlphaGo and its successors, have predominantly relied on model-based Monte-Carlo Tree Search (MCTS). In this work, we present QZero, a novel model-free reinforcement learning algorithm that forgoes search during training and learns a Nash equilibrium policy through self-play and off-policy experience replay. Built upon entropy-regularized Q-learning, QZero utilizes a single Q-value network to unify policy evaluation and improvement. Starting tabula rasa without human data and trained for 5 months with modest compute resources (7 GPUs), QZero achieved a performance level comparable to that of AlphaGo. This demonstrates, for the first time, the efficiency of using model-free reinforcement learning to master the game of Go, as well as the feasibility of off-policy reinforcement learning in solving large-scale and complex environments.