QMNet: Importance-Aware Message Exchange for Decentralized Multi-Agent Reinforcement Learning

TL;DR

This paper introduces QMNet, an importance-aware message exchange framework for decentralized multi-agent reinforcement learning that optimizes communication efficiency and system performance under limited wireless resources.

Contribution

The paper proposes a novel importance metric and scheduling policy, along with a query-message architecture and message prediction, to enhance multi-agent RL with limited communication resources.

Findings

QMNet maintains high performance with only 30% message sharing.

Message prediction reduces wireless resource usage by 40%.

Importance-aware access avoids collisions, matching centralized scheduling performance.

Abstract

To improve the performance of multi-agent reinforcement learning under the constraint of wireless resources, we propose a message importance metric and design an importance-aware scheduling policy to effectively exchange messages. The key insight is spending the precious communication resources on important messages. The message importance depends not only on the messages themselves, but also on the needs of agents who receive them. Accordingly, we propose a query-message-based architecture, called QMNet. Agents generate queries and messages with the environment observation. Sharing queries can help calculate message importance. Exchanging messages can help agents cooperate better. Besides, we exploit the message importance to deal with random access collisions in decentralized systems. Furthermore, a message prediction mechanism is proposed to compensate for messages that are not transmitted. Finally, we evaluate the proposed schemes in a traffic junction environment, where only a fraction of agents can send messages due to limited wireless resources. Results show that QMNet can extract valuable information to guarantee the system performance even when only $30\%$ of agents can share messages. By exploiting message prediction, the system can further save $40\%$ of wireless resources. The importance-aware decentralized multi-access mechanism can effectively avoid collisions, achieving almost the same performance as centralized scheduling.