Reproducing AlphaZero on Tablut: Self-Play RL for an Asymmetric Board Game

TL;DR

This paper adapts the AlphaZero reinforcement learning algorithm to the asymmetric board game Tablut by modifying its architecture and stabilization techniques, enabling effective self-play learning in a complex environment.

Contribution

The study introduces separate policy and value heads for each player role in AlphaZero, along with stabilization methods, to successfully apply it to asymmetric games like Tablut.

Findings

Achieved a BayesElo rating of 1235 after 100 self-play iterations.

Significant reduction in policy entropy indicating more decisive play.

Overcame training instabilities using data augmentation and replay buffer enhancements.

Abstract

This work investigates the adaptation of the AlphaZero reinforcement learning algorithm to Tablut, an asymmetric historical board game featuring unequal piece counts and distinct player objectives (king capture versus king escape). While the original AlphaZero architecture successfully leverages a single policy and value head for symmetric games, applying it to asymmetric environments forces the network to learn two conflicting evaluation functions, which can hinder learning efficiency and performance. To address this, the core architecture is modified to use separate policy and value heads for each player role, while maintaining a shared residual trunk to learn common board features. During training, the asymmetric structure introduced training instabilities, notably catastrophic forgetting between the attacker and defender roles. These issues were mitigated by applying C4 data augmentation, increasing the replay buffer size, and having the model play 25 percent of training games against randomly sampled past checkpoints. Over 100 self-play iterations, the modified model demonstrated steady improvement, achieving a BayesElo rating of 1235 relative to a randomly initialized baseline. Training metrics also showed a significant decrease in policy entropy and average remaining pieces, reflecting increasingly focused and decisive play. Ultimately, the experiments confirm that AlphaZero's self-play framework can transfer to highly asymmetric games, provided that distinct policy/value heads and robust stabilization techniques are employed.