MADiff: Offline Multi-agent Learning with Diffusion Models

TL;DR

MADiff introduces a novel diffusion model framework for offline multi-agent reinforcement learning, effectively capturing complex interactions and coordination among agents, outperforming existing methods in various tasks.

Contribution

It is the first diffusion-based multi-agent learning framework, combining decentralized policies with a centralized controller for improved coordination.

Findings

MADiff outperforms baseline algorithms in multiple multi-agent tasks.

The framework effectively models complex multi-agent interactions.

MADiff demonstrates both decentralized policy execution and centralized trajectory prediction.

Abstract

Offline reinforcement learning (RL) aims to learn policies from pre-existing datasets without further interactions, making it a challenging task. Q-learning algorithms struggle with extrapolation errors in offline settings, while supervised learning methods are constrained by model expressiveness. Recently, diffusion models (DMs) have shown promise in overcoming these limitations in single-agent learning, but their application in multi-agent scenarios remains unclear. Generating trajectories for each agent with independent DMs may impede coordination, while concatenating all agents' information can lead to low sample efficiency. Accordingly, we propose MADiff, which is realized with an attention-based diffusion model to model the complex coordination among behaviors of multiple agents. To our knowledge, MADiff is the first diffusion-based multi-agent learning framework, functioning as both a decentralized policy and a centralized controller. During decentralized executions, MADiff simultaneously performs teammate modeling, and the centralized controller can also be applied in multi-agent trajectory predictions. Our experiments demonstrate that MADiff outperforms baseline algorithms across various multi-agent learning tasks, highlighting its effectiveness in modeling complex multi-agent interactions. Our code is available at https://github.com/zbzhu99/madiff.