Exponential Topology-enabled Scalable Communication in Multi-agent Reinforcement Learning

TL;DR

This paper introduces ExpoComm, a scalable multi-agent communication protocol using exponential topology to improve information sharing and task performance in large-scale cooperative reinforcement learning.

Contribution

It proposes a novel exponential topology-based communication protocol for scalable multi-agent reinforcement learning, with memory-based message processing and auxiliary tasks.

Findings

Outperforms existing communication strategies on large-scale benchmarks

Demonstrates robust zero-shot transferability

Enables rapid information dissemination among agents

Abstract

In cooperative multi-agent reinforcement learning (MARL), well-designed communication protocols can effectively facilitate consensus among agents, thereby enhancing task performance. Moreover, in large-scale multi-agent systems commonly found in real-world applications, effective communication plays an even more critical role due to the escalated challenge of partial observability compared to smaller-scale setups. In this work, we endeavor to develop a scalable communication protocol for MARL. Unlike previous methods that focus on selecting optimal pairwise communication links-a task that becomes increasingly complex as the number of agents grows-we adopt a global perspective on communication topology design. Specifically, we propose utilizing the exponential topology to enable rapid information dissemination among agents by leveraging its small-diameter and small-size properties. This approach leads to a scalable communication protocol, named ExpoComm. To fully unlock the potential of exponential graphs as communication topologies, we employ memory-based message processors and auxiliary tasks to ground messages, ensuring that they reflect global information and benefit decision-making. Extensive experiments on large-scale cooperative benchmarks, including MAgent and Infrastructure Management Planning, demonstrate the superior performance and robust zero-shot transferability of ExpoComm compared to existing communication strategies. The code is publicly available at https://github.com/LXXXXR/ExpoComm.