Reward-Independent Messaging for Decentralized Multi-Agent Reinforcement Learning

TL;DR

This paper introduces MARL-CPC, a novel framework for decentralized multi-agent reinforcement learning that enables reward-independent communication through collective predictive coding, improving coordination without shared parameters.

Contribution

It presents a new message learning approach based on collective predictive coding that supports non-cooperative, reward-independent communication in decentralized MARL settings.

Findings

MARL-CPC outperforms standard message-as-action methods in benchmarks.

Effective communication is achieved even when messages do not benefit the sender.

Supports coordination in complex, decentralized environments.

Abstract

In multi-agent reinforcement learning (MARL), effective communication improves agent performance, particularly under partial observability. We propose MARL-CPC, a framework that enables communication among fully decentralized, independent agents without parameter sharing. MARL-CPC incorporates a message learning model based on collective predictive coding (CPC) from emergent communication research. Unlike conventional methods that treat messages as part of the action space and assume cooperation, MARL-CPC links messages to state inference, supporting communication in non-cooperative, reward-independent settings. We introduce two algorithms -Bandit-CPC and IPPO-CPC- and evaluate them in non-cooperative MARL tasks. Benchmarks show that both outperform standard message-as-action approaches, establishing effective communication even when messages offer no direct benefit to the sender. These results highlight MARL-CPC's potential for enabling coordination in complex, decentralized environments.