MA-Dreamer: Coordination and communication through shared imagination

TL;DR

MA-Dreamer is a model-based multi-agent reinforcement learning method that enables decentralized agents to coordinate and communicate effectively through shared imagination, outperforming existing approaches in complex, partially observable environments.

Contribution

It introduces a novel model-based approach using shared environment models for training decentralized policies and critics via imagination, improving coordination and communication in multi-agent settings.

Findings

MA-Dreamer outperforms other methods in soccer-based cooperative tasks.

It effectively handles long-term speaker-listener and partial-observability challenges.

The method demonstrates robust coordination and communication capabilities.

Abstract

Multi-agent RL is rendered difficult due to the non-stationary nature of environment perceived by individual agents. Theoretically sound methods using the REINFORCE estimator are impeded by its high-variance, whereas value-function based methods are affected by issues stemming from their ad-hoc handling of situations like inter-agent communication. Methods like MADDPG are further constrained due to their requirement of centralized critics etc. In order to address these issues, we present MA-Dreamer, a model-based method that uses both agent-centric and global differentiable models of the environment in order to train decentralized agents' policies and critics using model-rollouts a.k.a `imagination'. Since only the model-training is done off-policy, inter-agent communication/coordination and `language emergence' can be handled in a straight-forward manner. We compare the performance of MA-Dreamer with other methods on two soccer-based games. Our experiments show that in long-term speaker-listener tasks and in cooperative games with strong partial-observability, MA-Dreamer finds a solution that makes effective use of coordination, whereas competing methods obtain marginal scores and fail outright, respectively. By effectively achieving coordination and communication under more relaxed and general conditions, out method opens the door to the study of more complex problems and population-based training.