Offline Multi-agent Reinforcement Learning via Score Decomposition

TL;DR

This paper introduces a novel score decomposition method combined with diffusion models to improve offline multi-agent reinforcement learning, effectively addressing distributional shifts and multimodal joint behaviors, leading to state-of-the-art results.

Contribution

It proposes a new sequential score function decomposition technique and diffusion-based generative modeling to enhance offline MARL performance and generalization.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Effectively handles multimodal joint behavior data.

Addresses distributional gaps between offline and online MARL.

Abstract

Offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts, particularly stemming from the high dimensionality of joint action spaces and the presence of out-of-distribution joint action selections. In this work, we highlight that a fundamental challenge in offline MARL arises from the multi-equilibrium nature of cooperative tasks, which induces a highly multimodal joint behavior policy space coupled with heterogeneous-quality behavior data. This makes it difficult for individual policy regularization to align with a consistent coordination pattern, leading to the policy distribution shift problems. To tackle this challenge, we design a sequential score function decomposition method that distills per-agent regularization signals from the joint behavior policy, which induces coordinated modality selection under decentralized execution constraints. Then we leverage a flexible diffusion-based generative model to learn these score functions from multimodal offline data, and integrate them into joint-action critics to guide policy updates toward high-reward, in-distribution regions under a shared team reward. Our approach achieves state-of-the-art performance across multiple particle environments and Multi-agent MuJoCo benchmarks consistently. To the best of our knowledge, this is the first work to explicitly address the distributional gap between offline and online MARL, paving the way for more generalizable offline policy-based MARL methods.