MACRPO: Multi-Agent Cooperative Recurrent Policy Optimization

TL;DR

MACRPO introduces a novel multi-agent reinforcement learning method that enhances cooperation and information sharing through recurrent networks and a new advantage function, outperforming existing algorithms in complex environments.

Contribution

This paper presents MACRPO, a new multi-agent actor-critic algorithm with recurrent layers and a novel advantage function, improving cooperation in partially observable, non-stationary settings.

Findings

MACRPO outperforms state-of-the-art algorithms like QMIX and MADDPG.

Recurrent critic with meta-trajectory training improves cooperation.

Incorporating other agents' rewards enhances learning efficiency.

Abstract

This work considers the problem of learning cooperative policies in multi-agent settings with partially observable and non-stationary environments without a communication channel. We focus on improving information sharing between agents and propose a new multi-agent actor-critic method called \textit{Multi-Agent Cooperative Recurrent Proximal Policy Optimization} (MACRPO). We propose two novel ways of integrating information across agents and time in MACRPO: First, we use a recurrent layer in critic's network architecture and propose a new framework to use a meta-trajectory to train the recurrent layer. This allows the network to learn the cooperation and dynamics of interactions between agents, and also handle partial observability. Second, we propose a new advantage function that incorporates other agents' rewards and value functions. We evaluate our algorithm on three challenging multi-agent environments with continuous and discrete action spaces, Deepdrive-Zero, Multi-Walker, and Particle environment. We compare the results with several ablations and state-of-the-art multi-agent algorithms such as QMIX and MADDPG and also single-agent methods with shared parameters between agents such as IMPALA and APEX. The results show superior performance against other algorithms. The code is available online at https://github.com/kargarisaac/macrpo.