Neural Fictitious Self-Play on ELF Mini-RTS

TL;DR

This paper demonstrates that Neural Fictitious Self-Play (NFSP), combined with policy gradient reinforcement learning, can effectively address the multi-agent challenges in RTS games like Mini-RTS, improving scalability and strategy quality.

Contribution

It introduces applying NFSP to RTS games, integrating it with policy gradient methods, and enhancing scalability through pretraining with self-play.

Findings

NFSP can be combined with policy gradient reinforcement learning for RTS.

Pretraining with self-play improves NFSP scalability.

NFSP achieves strong strategies despite lack of convergence guarantees.

Abstract

Despite the notable successes in video games such as Atari 2600, current AI is yet to defeat human champions in the domain of real-time strategy (RTS) games. One of the reasons is that an RTS game is a multi-agent game, in which single-agent reinforcement learning methods cannot simply be applied because the environment is not a stationary Markov Decision Process. In this paper, we present a first step toward finding a game-theoretic solution to RTS games by applying Neural Fictitious Self-Play (NFSP), a game-theoretic approach for finding Nash equilibria, to Mini-RTS, a small but nontrivial RTS game provided on the ELF platform. More specifically, we show that NFSP can be effectively combined with policy gradient reinforcement learning and be applied to Mini-RTS. Experimental results also show that the scalability of NFSP can be substantially improved by pretraining the models with simple self-play using policy gradients, which by itself gives a strong strategy despite its lack of theoretical guarantee of convergence.