Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning

TL;DR

This paper introduces DDPL, a decentralized diffusion policy learning method using generative models to improve exploration in multi-agent reinforcement learning, addressing Gaussian policy limitations.

Contribution

Proposes DDPL with diffusion models for policies, enabling better exploration and efficient training via ISSM, outperforming Gaussian-based approaches.

Findings

DDPL achieves superior exploration in continuous-action MARL benchmarks.

Diffusion policies capture multi-modal action distributions effectively.

DDPL outperforms Gaussian policy methods across multiple environments.

Abstract

Cooperative multi-agent reinforcement learning (MARL) involves complex agent interactions and requires effective exploration strategies. A prominent class of MARL algorithms, decentralized softmax policy gradient (DecSPG), addresses this through energy-based policy updates. In practice, however, such energy-based policies are intractable to maintain and are commonly projected onto the Gaussian policy class. In this work, we show that the limited expressiveness of Gaussian policies severely hinders exploration in DecSPG, and this limitation worsens as the number of agents grows. To address this issue, we propose decentralized diffusion policy learning (DDPL), which parameterizes each agent's policy with a denoising diffusion probabilistic model, an expressive generative model that captures multi-modal action distributions for enhanced exploration. DDPL enables efficient online training of diffusion policies via importance sampling score matching (ISSM), a novel training method with theoretical guarantee. We evaluate DDPL on representative continuous-action MARL benchmarks, including multi-agent particle environment, multi-agent MuJoCo, IsaacLab, and JAX-reimplemented StarCraft multi-agent challenge, and observe consistently improved performance.