Minimizing Communication while Maximizing Performance in Multi-Agent Reinforcement Learning

TL;DR

This paper proposes a method to reduce communication in multi-agent reinforcement learning by optimizing a combined objective with a penalty, achieving up to 75% less communication without performance loss.

Contribution

It introduces a novel training approach with techniques like partial training and message forwarding to minimize communication while maintaining performance in multi-agent tasks.

Findings

Reduced communication by 75% without performance loss.

Training with a communication penalty on 50% of episodes stabilizes learning.

Message forwarding improves information retention and performance.

Abstract

Inter-agent communication can significantly increase performance in multi-agent tasks that require co-ordination to achieve a shared goal. Prior work has shown that it is possible to learn inter-agent communication protocols using multi-agent reinforcement learning and message-passing network architectures. However, these models use an unconstrained broadcast communication model, in which an agent communicates with all other agents at every step, even when the task does not require it. In real-world applications, where communication may be limited by system constraints like bandwidth, power and network capacity, one might need to reduce the number of messages that are sent. In this work, we explore a simple method of minimizing communication while maximizing performance in multi-task learning: simultaneously optimizing a task-specific objective and a communication penalty. We show that the objectives can be optimized using Reinforce and the Gumbel-Softmax reparameterization. We introduce two techniques to stabilize training: 50% training and message forwarding. Training with the communication penalty on only 50% of the episodes prevents our models from turning off their outgoing messages. Second, repeating messages received previously helps models retain information, and further improves performance. With these techniques, we show that we can reduce communication by 75% with no loss of performance.