A Survey on Self-play Methods in Reinforcement Learning

TL;DR

This survey provides a comprehensive overview of self-play methods in reinforcement learning, categorizing algorithms, discussing their applications in multi-agent tasks, and outlining future research challenges.

Contribution

It offers a unified framework for understanding self-play algorithms and bridges the gap between theory and practical applications in multi-agent reinforcement learning.

Findings

Classifies existing self-play algorithms within a unified framework

Highlights the role of self-play in complex multi-agent tasks like Go and poker

Identifies open challenges and future directions in self-play research

Abstract

Self-play, a learning paradigm where agents iteratively refine their policies by interacting with historical or concurrent versions of themselves or other evolving agents, has shown remarkable success in solving complex non-cooperative multi-agent tasks. Despite its growing prominence in multi-agent reinforcement learning (MARL), such as Go, poker, and video games, a comprehensive and structured understanding of self-play remains lacking. This survey fills this gap by offering a comprehensive roadmap to the diverse landscape of self-play methods. We begin by introducing the necessary preliminaries, including the MARL framework and basic game theory concepts. Then, it provides a unified framework and classifies existing self-play algorithms within this framework. Moreover, the paper bridges the gap between the algorithms and their practical implications by illustrating the role of self-play in different non-cooperative scenarios. Finally, the survey highlights open challenges and future research directions in self-play.