Efficient Competitive Self-Play Policy Optimization

TL;DR

This paper introduces a new competitive self-play reinforcement learning framework that leverages saddle point optimization principles, leading to improved convergence and performance in various two-player zero-sum games.

Contribution

The paper proposes a novel algorithmic framework for competitive self-play that uses saddle point optimization ideas, ensuring convergence and empirical superiority over heuristic opponent selection methods.

Findings

Proves convergence to an approximate equilibrium in convex-concave games.

Demonstrates empirical improvements in matrix games, grid-world soccer, Gomoku, and robot sumo.

Outperforms baseline opponent-selection heuristics in various environments.

Abstract

Reinforcement learning from self-play has recently reported many successes. Self-play, where the agents compete with themselves, is often used to generate training data for iterative policy improvement. In previous work, heuristic rules are designed to choose an opponent for the current learner. Typical rules include choosing the latest agent, the best agent, or a random historical agent. However, these rules may be inefficient in practice and sometimes do not guarantee convergence even in the simplest matrix games. In this paper, we propose a new algorithmic framework for competitive self-play reinforcement learning in two-player zero-sum games. We recognize the fact that the Nash equilibrium coincides with the saddle point of the stochastic payoff function, which motivates us to borrow ideas from classical saddle point optimization literature. Our method trains several agents simultaneously, and intelligently takes each other as opponent based on simple adversarial rules derived from a principled perturbation-based saddle optimization method. We prove theoretically that our algorithm converges to an approximate equilibrium with high probability in convex-concave games under standard assumptions. Beyond the theory, we further show the empirical superiority of our method over baseline methods relying on the aforementioned opponent-selection heuristics in matrix games, grid-world soccer, Gomoku, and simulated robot sumo, with neural net policy function approximators.