Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learning

TL;DR

This paper introduces MASIA, a self-supervised message aggregation method for multi-agent reinforcement learning that improves coordination by creating compact, relevant message representations, and also establishes offline benchmarks for communication research.

Contribution

The paper proposes a novel self-supervised message aggregation technique for cooperative MARL and creates the first offline benchmarks for multi-agent communication.

Findings

MASIA outperforms existing methods in online settings.

The offline benchmarks enable new research directions.

Empirical results validate the effectiveness of message aggregation.

Abstract

Utilizing messages from teammates can improve coordination in cooperative Multi-agent Reinforcement Learning (MARL). Previous works typically combine raw messages of teammates with local information as inputs for policy. However, neglecting message aggregation poses significant inefficiency for policy learning. Motivated by recent advances in representation learning, we argue that efficient message aggregation is essential for good coordination in cooperative MARL. In this paper, we propose Multi-Agent communication via Self-supervised Information Aggregation (MASIA), where agents can aggregate the received messages into compact representations with high relevance to augment the local policy. Specifically, we design a permutation invariant message encoder to generate common information-aggregated representation from messages and optimize it via reconstructing and shooting future information in a self-supervised manner. Hence, each agent would utilize the most relevant parts of the aggregated representation for decision-making by a novel message extraction mechanism. Furthermore, considering the potential of offline learning for real-world applications, we build offline benchmarks for multi-agent communication, which is the first as we know. Empirical results demonstrate the superiority of our method in both online and offline settings. We also release the built offline benchmarks in this paper as a testbed for communication ability validation to facilitate further future research.