MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning

TL;DR

MALib is a scalable, efficient parallel framework designed for population-based multi-agent reinforcement learning, enabling high throughput and faster training across complex multi-agent tasks.

Contribution

The paper introduces MALib, a novel framework that addresses parallelization challenges in PB-MARL through centralized task dispatching, a new Actor-Evaluator-Learner architecture, and flexible training abstractions.

Findings

Achieves over 40K FPS on a single machine with 32 CPU cores.

Provides 5x speedup over RLlib in multi-agent training.

Offers at least 3x speedup compared to OpenSpiel.

Abstract

Population-based multi-agent reinforcement learning (PB-MARL) refers to the series of methods nested with reinforcement learning (RL) algorithms, which produces a self-generated sequence of tasks arising from the coupled population dynamics. By leveraging auto-curricula to induce a population of distinct emergent strategies, PB-MARL has achieved impressive success in tackling multi-agent tasks. Despite remarkable prior arts of distributed RL frameworks, PB-MARL poses new challenges for parallelizing the training frameworks due to the additional complexity of multiple nested workloads between sampling, training and evaluation involved with heterogeneous policy interactions. To solve these problems, we present MALib, a scalable and efficient computing framework for PB-MARL. Our framework is comprised of three key components: (1) a centralized task dispatching model, which supports the self-generated tasks and scalable training with heterogeneous policy combinations; (2) a programming architecture named Actor-Evaluator-Learner, which achieves high parallelism for both training and sampling, and meets the evaluation requirement of auto-curriculum learning; (3) a higher-level abstraction of MARL training paradigms, which enables efficient code reuse and flexible deployments on different distributed computing paradigms. Experiments on a series of complex tasks such as multi-agent Atari Games show that MALib achieves throughput higher than 40K FPS on a single machine with $32$ CPU cores; 5x speedup than RLlib and at least 3x speedup than OpenSpiel in multi-agent training tasks. MALib is publicly available at https://github.com/sjtu-marl/malib.