Impartial Games: A Challenge for Reinforcement Learning

TL;DR

This paper demonstrates that AlphaZero-style reinforcement learning algorithms face fundamental challenges in mastering impartial games like Nim, due to their inability to learn abstract mathematical principles such as parity, especially as game complexity increases.

Contribution

The paper introduces a new framework to evaluate RL agents in impartial games and reveals inherent representational limitations of neural networks in learning abstract functions like parity.

Findings

AlphaZero-style agents succeed on small Nim boards but struggle as size increases.

Neural networks have difficulty learning non-associative functions like parity.

Current RL algorithms cannot effectively master impartial games beyond rote memorization.

Abstract

AlphaZero-style reinforcement learning (RL) algorithms have achieved superhuman performance in many complex board games such as Chess, Shogi, and Go. However, we showcase that these algorithms encounter significant and fundamental challenges when applied to impartial games, a class where players share game pieces and optimal strategy often relies on abstract mathematical principles. Specifically, we utilise the game of Nim as a concrete and illustrative case study to reveal critical limitations of AlphaZero-style and similar self-play RL algorithms. We introduce a novel conceptual framework distinguishing between champion and expert mastery to evaluate RL agent performance. Our findings reveal that while AlphaZero-style agents can achieve champion-level play on very small Nim boards, their learning progression severely degrades as the board size increases. This difficulty stems not merely from complex data distributions or noisy labels, but from a deeper representational bottleneck: the inherent struggle of generic neural networks to implicitly learn abstract, non-associative functions like parity, which are crucial for optimal play in impartial games. This limitation causes a critical breakdown in the positive feedback loop essential for self-play RL, preventing effective learning beyond rote memorisation of frequently observed states. These results align with broader concerns regarding AlphaZero-style algorithms' vulnerability to adversarial attacks, highlighting their inability to truly master all legal game states. Our work underscores that simple hyperparameter adjustments are insufficient to overcome these challenges, establishing a crucial foundation for the development of fundamentally novel algorithmic approaches, potentially involving neuro-symbolic or meta-learning paradigms, to bridge the gap towards true expert-level AI in combinatorial games.