Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Games

TL;DR

This paper analyzes a regularized policy optimization method for two-player zero-sum games, providing theoretical convergence guarantees and demonstrating improved training efficiency in various board games.

Contribution

It offers new theoretical convergence guarantees and develops a practical reinforcement learning algorithm that outperforms existing methods in multiple two-player games.

Findings

The policy update rule is stable with convergence guarantees in theoretical settings.

The proposed algorithm learns more efficiently than existing methods.

Empirical validation on five board games shows improved training efficiency.

Abstract

Two-player games such as board games have long been used as traditional benchmarks for reinforcement learning. This work revisits a policy optimization method with reverse Kullback-Leibler regularization and entropy regularization and analyzes this combination in two-player zero-sum settings from theoretical and empirical perspectives. From a theoretical perspective, we investigate the stability of the policy update rule in two theoretical settings: game-theoretic normal-form games and finite-length games. We provide novel convergence guarantees and verify our theoretical results through numerical experiments on synthetic games. From an empirical perspective, we derive a practical model-free reinforcement learning algorithm based on the regularized policy optimization. We validate the training efficiency of our algorithm through comprehensive experiments on five board games: Animal Shogi, Gardner Chess, Go, Hex, and Othello. Experimental results show that our agent learns more efficiently than existing methods across environments.