Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces asynchronous multi-agent actor-critic algorithms enabling agents to learn and act independently without synchronization, improving performance in complex multi-agent environments.

Contribution

It develops the first set of asynchronous policy gradient methods for multi-agent reinforcement learning across various training paradigms.

Findings

Algorithms outperform synchronous methods in large multi-agent tasks.

Empirical validation in simulation and hardware shows high-quality asynchronous solutions.

Methods are effective across diverse realistic domains.

Abstract

Synchronizing decisions across multiple agents in realistic settings is problematic since it requires agents to wait for other agents to terminate and communicate about termination reliably. Ideally, agents should learn and execute asynchronously instead. Such asynchronous methods also allow temporally extended actions that can take different amounts of time based on the situation and action executed. Unfortunately, current policy gradient methods are not applicable in asynchronous settings, as they assume that agents synchronously reason about action selection at every time step. To allow asynchronous learning and decision-making, we formulate a set of asynchronous multi-agent actor-critic methods that allow agents to directly optimize asynchronous policies in three standard training paradigms: decentralized learning, centralized learning, and centralized training for decentralized execution. Empirical results (in simulation and hardware) in a variety of realistic domains demonstrate the superiority of our approaches in large multi-agent problems and validate the effectiveness of our algorithms for learning high-quality and asynchronous solutions.