Mastering Chinese Chess AI (Xiangqi) Without Search

TL;DR

This paper introduces a high-performance Chinese Chess AI that operates without search algorithms, achieving top-tier human-level play through a novel training system combining supervised and reinforcement learning.

Contribution

The paper presents a search-free Chinese Chess AI utilizing Transformer architecture and innovative training techniques, surpassing traditional search-based methods in speed and strength.

Findings

Transformer outperforms CNN in Chinese chess performance

Move features and opponent pools enhance training efficiency

VECT improves PPO training process

Abstract

We have developed a high-performance Chinese Chess AI that operates without reliance on search algorithms. This AI has demonstrated the capability to compete at a level commensurate with the top 0.1\% of human players. By eliminating the search process typically associated with such systems, this AI achieves a Queries Per Second (QPS) rate that exceeds those of systems based on the Monte Carlo Tree Search (MCTS) algorithm by over a thousandfold and surpasses those based on the AlphaBeta pruning algorithm by more than a hundredfold. The AI training system consists of two parts: supervised learning and reinforcement learning. Supervised learning provides an initial human-like Chinese chess AI, while reinforcement learning, based on supervised learning, elevates the strength of the entire AI to a new level. Based on this training system, we carried out enough ablation experiments and discovered that 1. The same parameter amount of Transformer architecture has a higher performance than CNN on Chinese chess; 2. Possible moves of both sides as features can greatly improve the training process; 3. Selective opponent pool, compared to pure self-play training, results in a faster improvement curve and a higher strength limit. 4. Value Estimation with Cutoff(VECT) improves the original PPO algorithm training process and we will give the explanation.