Containerized Distributed Value-Based Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a containerized distributed multi-agent reinforcement learning framework that enhances scalability, diversity, and efficiency, successfully applying it to complex environments like Google Research Football and StarCraft II.

Contribution

It presents a novel containerized architecture with multi-queue management for scalable, diverse, and efficient distributed MARL, achieving state-of-the-art results in complex benchmarks.

Findings

Achieved 4-18x better results than non-distributed MARL algorithms on StarCraft II.

First to successfully solve Google Research Football full game 5v5.

Developed a scalable, time-efficient distributed MARL framework.

Abstract

Multi-agent reinforcement learning tasks put a high demand on the volume of training samples. Different from its single-agent counterpart, distributed value-based multi-agent reinforcement learning faces the unique challenges of demanding data transfer, inter-process communication management, and high requirement of exploration. We propose a containerized learning framework to solve these problems. We pack several environment instances, a local learner and buffer, and a carefully designed multi-queue manager which avoids blocking into a container. Local policies of each container are encouraged to be as diverse as possible, and only trajectories with highest priority are sent to a global learner. In this way, we achieve a scalable, time-efficient, and diverse distributed MARL learning framework with high system throughput. To own knowledge, our method is the first to solve the challenging Google Research Football full game $5\_v\_5$. On the StarCraft II micromanagement benchmark, our method gets $4$-$18\times$ better results compared to state-of-the-art non-distributed MARL algorithms.