Learning what to say and how precisely: Efficient Communication via Differentiable Discrete Communication Learning

TL;DR

This paper introduces an extension of Differentiable Discrete Communication Learning (DDCL) that supports unbounded signals, enabling efficient, adaptive communication in multi-agent reinforcement learning with reduced bandwidth and competitive performance.

Contribution

The paper extends DDCL to support unbounded signals, creating a universal layer for MARL that dynamically modulates message precision and reduces bandwidth without sacrificing performance.

Findings

Agents learn to adapt message precision based on task needs.

Bandwidth is reduced by over an order of magnitude while maintaining or improving performance.

A simple Transformer-based policy with DDCL matches complex architectures, questioning the need for specialized designs.

Abstract

Effective communication in multi-agent reinforcement learning (MARL) is critical for success but constrained by bandwidth, yet past approaches have been limited to complex gating mechanisms that only decide \textit{whether} to communicate, not \textit{how precisely}. Learning to optimize message precision at the bit-level is fundamentally harder, as the required discretization step breaks gradient flow. We address this by generalizing Differentiable Discrete Communication Learning (DDCL), a framework for end-to-end optimization of discrete messages. Our primary contribution is an extension of DDCL to support unbounded signals, transforming it into a universal, plug-and-play layer for any MARL architecture. We verify our approach with three key results. First, through a qualitative analysis in a controlled environment, we demonstrate \textit{how} agents learn to dynamically modulate message precision according to the informational needs of the task. Second, we integrate our variant of DDCL into four state-of-the-art MARL algorithms, showing it reduces bandwidth by over an order of magnitude while matching or exceeding task performance. Finally, we provide direct evidence for the \enquote{Bitter Lesson} in MARL communication: a simple Transformer-based policy leveraging DDCL matches the performance of complex, specialized architectures, questioning the necessity of bespoke communication designs.