Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning

TL;DR

This paper introduces SACL, a novel subgame curriculum learning framework that accelerates multi-agent reinforcement learning in zero-sum games by adaptively selecting subgames, leading to faster convergence and stronger policies.

Contribution

The paper proposes a new subgame curriculum learning method, SACL, which uses an adaptive initial state distribution and a subgame selection metric to improve MARL efficiency in zero-sum games.

Findings

SACL outperforms baseline algorithms in particle-world and Google Research Football environments.

SACL achieves emergent stages and reduces sample complexity in hide-and-seek quadrants.

SACL enhances policy strength and learning speed in complex zero-sum games.

Abstract

Learning Nash equilibrium (NE) in complex zero-sum games with multi-agent reinforcement learning (MARL) can be extremely computationally expensive. Curriculum learning is an effective way to accelerate learning, but an under-explored dimension for generating a curriculum is the difficulty-to-learn of the subgames -- games induced by starting from a specific state. In this work, we present a novel subgame curriculum learning framework for zero-sum games. It adopts an adaptive initial state distribution by resetting agents to some previously visited states where they can quickly learn to improve performance. Building upon this framework, we derive a subgame selection metric that approximates the squared distance to NE values and further adopt a particle-based state sampler for subgame generation. Integrating these techniques leads to our new algorithm, Subgame Automatic Curriculum Learning (SACL), which is a realization of the subgame curriculum learning framework. SACL can be combined with any MARL algorithm such as MAPPO. Experiments in the particle-world environment and Google Research Football environment show SACL produces much stronger policies than baselines. In the challenging hide-and-seek quadrant environment, SACL produces all four emergent stages and uses only half the samples of MAPPO with self-play. The project website is at https://sites.google.com/view/sacl-rl.