CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning

TL;DR

CODA introduces a diffusion-based data augmentation method for multi-agent offline reinforcement learning, enabling better coordination by reflecting evolving agent behaviors during training.

Contribution

The paper presents CODA, a novel on-policy diffusion-based trajectory generator that improves multi-agent coordination in offline RL by dynamically augmenting data.

Findings

CODA resolves coordination failures in continuous polynomial games.

CODA achieves strong results on MaMuJoCo benchmarks.

Previous static augmentation methods are insufficient for multi-agent coordination.

Abstract

Offline multi-agent reinforcement learning (MARL) enables policy learning from fixed datasets, but is prone to coordination failure: agents trained on static, off-policy data converge to suboptimal joint behaviours because they cannot co-adapt as their policies change. We introduce CODA (Coordination via On-Policy Diffusion for Multi-Agent Reinforcement Learning), a diffusion-based multi-agent trajectory generator for data augmentation that samples conditioned on the current joint policy, producing synthetic experience which reflects the evolving behaviours of the agents, thereby providing a mechanism for co-adaptation. We find that previous diffusion-based augmentation approaches are insufficient for fostering multi-agent coordination because they produce static augmented datasets that do not evolve as the current joint policy changes during training; CODA resolves this by more closely simulating on-policy learning and is a meaningful step toward coordinated behaviours in the offline setting. CODA is algorithm-agnostic and can be layered onto both model-free and model-based offline reinforcement learning pipelines as an augmentation module. Empirically, CODA not only resolves canonical coordination pathologies in continuous polynomial games but also delivers strong results on the more complex MaMuJoCo continuous-control benchmarks.