Multi-Agent Deep Reinforcement Learning Under Constrained Communications

TL;DR

This paper introduces a fully distributed multi-agent reinforcement learning framework that eliminates the need for centralized information, enabling scalable, robust, and adaptable cooperation among agents using local observations and multi-hop communication.

Contribution

The paper proposes a novel Distributed Graph Attention Network (D-GAT) and a distributed MARL framework called DG-MAPPO that operate without centralized critics or global state information.

Findings

Outperforms strong CTDE baselines on multiple multi-agent benchmarks

Achieves superior coordination with both homogeneous and heterogeneous teams

Provides a scalable and robust decentralized learning approach

Abstract

Centralized training with decentralized execution (CTDE) has been the dominant paradigm in multi-agent reinforcement learning (MARL), but its reliance on global state information during training introduces scalability, robustness, and generalization bottlenecks. Moreover, in practical scenarios such as adding/dropping teammates or facing environment dynamics that differ from the training, CTDE methods can be brittle and costly to retrain, whereas distributed approaches allow agents to adapt using only local information and peer-to-peer communication. We present a distributed MARL framework that removes the need for centralized critics or global information. Firstly, we develop a novel Distributed Graph Attention Network (D-GAT) that performs global state inference through multi-hop communication, where agents integrate neighbor features via input-dependent attention weights in a fully distributed manner. Leveraging D-GAT, we develop the distributed graph-attention MAPPO (DG-MAPPO) -- a distributed MARL framework where agents optimize local policies and value functions using local observations, multi-hop communication, and shared/averaged rewards. Empirical evaluation on the StarCraftII Multi-Agent Challenge, Google Research Football, and Multi-Agent Mujoco demonstrates that our method consistently outperforms strong CTDE baselines, achieving superior coordination across a wide range of cooperative tasks with both homogeneous and heterogeneous teams. Our distributed MARL framework provides a principled and scalable solution for robust collaboration, eliminating the need for centralized training or global observability. To the best of our knowledge, DG-MAPPO appears to be the first to fully eliminate reliance on privileged centralized information, enabling agents to learn and act solely through peer-to-peer communication.