Safe Continuous-time Multi-Agent Reinforcement Learning via Epigraph Form

TL;DR

This paper introduces a novel continuous-time multi-agent reinforcement learning framework that incorporates safety constraints using an epigraph reformulation and PINN-based optimization, improving stability and performance in complex environments.

Contribution

It proposes a new CT-CMDP formulation with an epigraph-based reformulation and a PINN-based actor-critic method for safe continuous-time MARL, addressing safety and stability issues.

Findings

Smoother value function approximations achieved.

More stable training observed in experiments.

Improved performance over existing safe MARL methods.

Abstract

Multi-agent reinforcement learning (MARL) has made significant progress in recent years, but most algorithms still rely on a discrete-time Markov Decision Process (MDP) with fixed decision intervals. This formulation is often ill-suited for complex multi-agent dynamics, particularly in high-frequency or irregular time-interval settings, leading to degraded performance and motivating the development of continuous-time MARL (CT-MARL). Existing CT-MARL methods are mainly built on Hamilton-Jacobi-Bellman (HJB) equations. However, they rarely account for safety constraints such as collision penalties, since these introduce discontinuities that make HJB-based learning difficult. To address this challenge, we propose a continuous-time constrained MDP (CT-CMDP) formulation and a novel MARL framework that transforms discrete MDPs into CT-CMDPs via an epigraph-based reformulation. We then solve this by proposing a novel physics-informed neural network (PINN)-based actor-critic method that enables stable and efficient optimization in continuous time. We evaluate our approach on continuous-time safe multi-particle environments (MPE) and safe multi-agent MuJoCo benchmarks. Results demonstrate smoother value approximations, more stable training, and improved performance over safe MARL baselines, validating the effectiveness and robustness of our method.