A Comparison of Self-Play Algorithms Under a Generalized Framework

TL;DR

This paper introduces a formal framework for understanding self-play in multiagent reinforcement learning, compares different algorithms within this framework, and analyzes their approximation to a theoretical solution in a simple environment.

Contribution

It provides the first formalized model of self-play, enabling systematic comparison and interpretation of self-play algorithms in multiagent RL.

Findings

Self-play algorithms exhibit cyclic policy evolutions during training.

The framework helps interpret performance metrics of self-play methods.

Different self-play definitions vary in how well they approximate the theoretical solution.

Abstract

Throughout scientific history, overarching theoretical frameworks have allowed researchers to grow beyond personal intuitions and culturally biased theories. They allow to verify and replicate existing findings, and to link is connected results. The notion of self-play, albeit often cited in multiagent Reinforcement Learning, has never been grounded in a formal model. We present a formalized framework, with clearly defined assumptions, which encapsulates the meaning of self-play as abstracted from various existing self-play algorithms. This framework is framed as an approximation to a theoretical solution concept for multiagent training. On a simple environment, we qualitatively measure how well a subset of the captured self-play methods approximate this solution when paired with the famous PPO algorithm. We also provide insights on interpreting quantitative metrics of performance for self-play training. Our results indicate that, throughout training, various self-play definitions exhibit cyclic policy evolutions.