Reaching Consensus in Cooperative Multi-Agent Reinforcement Learning with Goal Imagination

TL;DR

This paper introduces MAGI, a model-based framework that enables multi-agent systems to explicitly reach consensus through goal imagination, improving coordination, sample efficiency, and performance in cooperative reinforcement learning tasks.

Contribution

The paper proposes a novel goal imagination framework that models future state distributions to explicitly coordinate agents in multi-agent reinforcement learning.

Findings

MAGI improves sample efficiency in multi-agent environments.

MAGI outperforms existing methods in cooperative tasks.

The approach effectively guides agents to high-value future states.

Abstract

Reaching consensus is key to multi-agent coordination. To accomplish a cooperative task, agents need to coherently select optimal joint actions to maximize the team reward. However, current cooperative multi-agent reinforcement learning (MARL) methods usually do not explicitly take consensus into consideration, which may cause miscoordination problem. In this paper, we propose a model-based consensus mechanism to explicitly coordinate multiple agents. The proposed Multi-agent Goal Imagination (MAGI) framework guides agents to reach consensus with an Imagined common goal. The common goal is an achievable state with high value, which is obtained by sampling from the distribution of future states. We directly model this distribution with a self-supervised generative model, thus alleviating the "curse of dimensinality" problem induced by multi-agent multi-step policy rollout commonly used in model-based methods. We show that such efficient consensus mechanism can guide all agents cooperatively reaching valuable future states. Results on Multi-agent Particle-Environments and Google Research Football environment demonstrate the superiority of MAGI in both sample efficiency and performance.