Monte Carlo Neural Fictitious Self-Play: Approach to Approximate Nash equilibrium of Imperfect-Information Games

TL;DR

This paper introduces Monte Carlo Neural Fictitious Self Play (MC-NFSP), an advanced algorithm combining Monte Carlo tree search with NFSP to better approximate Nash equilibrium in large-scale imperfect-information games, outperforming previous methods.

Contribution

It proposes MC-NFSP, integrating Monte Carlo tree search with NFSP, and develops ANFSP with asynchronous parallel architecture to enhance convergence and training stability.

Findings

MC-NFSP converges to approximate Nash equilibrium in large-scale games.

MC-NFSP outperforms NFSP in deep search scenarios.

Parallel ANFSP accelerates training and stabilizes learning.

Abstract

Researchers on artificial intelligence have achieved human-level intelligence in large-scale perfect-information games, but it is still a challenge to achieve (nearly) optimal results (in other words, an approximate Nash Equilibrium) in large-scale imperfect-information games (i.e. war games, football coach or business strategies). Neural Fictitious Self Play (NFSP) is an effective algorithm for learning approximate Nash equilibrium of imperfect-information games from self-play without prior domain knowledge. However, it relies on Deep Q-Network, which is off-line and is hard to converge in online games with changing opponent strategy, so it can't approach approximate Nash equilibrium in games with large search scale and deep search depth. In this paper, we propose Monte Carlo Neural Fictitious Self Play (MC-NFSP), an algorithm combines Monte Carlo tree search with NFSP, which greatly improves the performance on large-scale zero-sum imperfect-information games. Experimentally, we demonstrate that the proposed Monte Carlo Neural Fictitious Self Play can converge to approximate Nash equilibrium in games with large-scale search depth while the Neural Fictitious Self Play can't. Furthermore, we develop Asynchronous Neural Fictitious Self Play (ANFSP). It use asynchronous and parallel architecture to collect game experience. In experiments, we show that parallel actor-learners have a further accelerated and stabilizing effect on training.