A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem

TL;DR

This paper introduces MOMA-PPO, a novel model-based offline multi-agent reinforcement learning algorithm that effectively addresses coordination challenges by generating synthetic data, outperforming existing model-free methods in complex, real-world scenarios.

Contribution

The paper presents the first model-based approach to offline MARL, overcoming coordination issues and demonstrating superior performance over model-free algorithms in challenging environments.

Findings

MOMA-PPO outperforms model-free methods in coordination tasks

Model-based methods handle partial observability better

Synthetic interaction data improves convergence and policy fine-tuning

Abstract

Training multiple agents to coordinate is an essential problem with applications in robotics, game theory, economics, and social sciences. However, most existing Multi-Agent Reinforcement Learning (MARL) methods are online and thus impractical for real-world applications in which collecting new interactions is costly or dangerous. While these algorithms should leverage offline data when available, doing so gives rise to what we call the offline coordination problem. Specifically, we identify and formalize the strategy agreement (SA) and the strategy fine-tuning (SFT) coordination challenges, two issues at which current offline MARL algorithms fail. Concretely, we reveal that the prevalent model-free methods are severely deficient and cannot handle coordination-intensive offline multi-agent tasks in either toy or MuJoCo domains. To address this setback, we emphasize the importance of inter-agent interactions and propose the very first model-based offline MARL method. Our resulting algorithm, Model-based Offline Multi-Agent Proximal Policy Optimization (MOMA-PPO) generates synthetic interaction data and enables agents to converge on a strategy while fine-tuning their policies accordingly. This simple model-based solution solves the coordination-intensive offline tasks, significantly outperforming the prevalent model-free methods even under severe partial observability and with learned world models.