Scalable Safe Multi-Agent Reinforcement Learning for Multi-Agent System

TL;DR

This paper introduces SS-MARL, a scalable and safe multi-agent reinforcement learning framework that uses graph-based message passing and constrained policy optimization to improve safety and scalability in large multi-agent systems.

Contribution

The paper proposes a novel graph-structured message passing network and constrained policy optimization for scalable and safe multi-agent reinforcement learning.

Findings

Outperforms baselines in safety and optimality trade-offs

Achieves superior scalability in large agent scenarios

Demonstrates effectiveness through simulation experiments

Abstract

Safety and scalability are two critical challenges faced by practical Multi-Agent Systems (MAS). However, existing Multi-Agent Reinforcement Learning (MARL) algorithms that rely solely on reward shaping are ineffective in ensuring safety, and their scalability is rather limited due to the fixed-size network output. To address these issues, we propose a novel framework, Scalable Safe MARL (SS-MARL), to enhance the safety and scalability of MARL methods. Leveraging the inherent graph structure of MAS, we design a multi-layer message passing network to aggregate local observations and communications of varying sizes. Furthermore, we develop a constrained joint policy optimization method in the setting of local observation to improve safety. Simulation experiments demonstrate that SS-MARL achieves a better trade-off between optimality and safety compared to baselines, and its scalability significantly outperforms the latest methods in scenarios with a large number of agents.