Low Entropy Communication in Multi-Agent Reinforcement Learning

TL;DR

This paper proposes a pseudo gradient descent method to significantly reduce message entropy in multi-agent reinforcement learning communication, enabling more efficient resource use without sacrificing cooperation performance.

Contribution

It introduces a novel pseudo gradient approach to decrease message entropy, addressing the challenge of non-gradient-friendly entropy reduction in multi-agent systems.

Findings

Reduced message entropy by up to 90%

Achieved minimal cooperation performance loss

Validated across multiple frameworks and environments

Abstract

Communication in multi-agent reinforcement learning has been drawing attention recently for its significant role in cooperation. However, multi-agent systems may suffer from limitations on communication resources and thus need efficient communication techniques in real-world scenarios. According to the Shannon-Hartley theorem, messages to be transmitted reliably in worse channels require lower entropy. Therefore, we aim to reduce message entropy in multi-agent communication. A fundamental challenge is that the gradients of entropy are either 0 or infinity, disabling gradient-based methods. To handle it, we propose a pseudo gradient descent scheme, which reduces entropy by adjusting the distributions of messages wisely. We conduct experiments on two base communication frameworks with six environment settings and find that our scheme can reduce message entropy by up to 90% with nearly no loss of cooperation performance.