Diffusion Models for Offline Multi-agent Reinforcement Learning with Safety Constraints

TL;DR

This paper introduces a novel diffusion model-based framework for offline multi-agent reinforcement learning that emphasizes safety and coordinated action, demonstrating superior performance and safety adherence on benchmark tasks.

Contribution

The paper presents a new diffusion model approach integrated with MARL under CTDE architecture, enhancing safety and coordination in offline settings.

Findings

Achieves safety compliance in multi-agent tasks

Outperforms existing methods on DSRL benchmark

Demonstrates effective risk mitigation in multi-agent coordination

Abstract

In recent advancements in Multi-agent Reinforcement Learning (MARL), its application has extended to various safety-critical scenarios. However, most methods focus on online learning, which presents substantial risks when deployed in real-world settings. Addressing this challenge, we introduce an innovative framework integrating diffusion models within the MARL paradigm. This approach notably enhances the safety of actions taken by multiple agents through risk mitigation while modeling coordinated action. Our framework is grounded in the Centralized Training with Decentralized Execution (CTDE) architecture, augmented by a Diffusion Model for prediction trajectory generation. Additionally, we incorporate a specialized algorithm to further ensure operational safety. We evaluate our model against baselines on the DSRL benchmark. Experiment results demonstrate that our model not only adheres to stringent safety constraints but also achieves superior performance compared to existing methodologies. This underscores the potential of our approach in advancing the safety and efficacy of MARL in real-world applications.