Mean-Field Approximation of Cooperative Constrained Multi-Agent Reinforcement Learning (CMARL)

TL;DR

This paper extends mean-field control methods to constrained multi-agent reinforcement learning, providing theoretical error bounds and a natural policy gradient algorithm with sample complexity guarantees.

Contribution

It introduces a mean-field approximation framework for constrained MARL and develops a natural policy gradient algorithm with proven error bounds and sample complexity.

Findings

Error bound of order $rac{ig[ ext{sqrt}| ext{X}| + ext{sqrt}| ext{U}|ig]}{ ext{sqrt}N}$ for constrained MARL approximation.

Improved error bound of order $rac{ ext{sqrt}| ext{X}|}{ ext{sqrt}N}$ in special cases.

Proposed algorithm achieves an error of $ ext{O}(e)$ with sample complexity $ ext{O}(e^{-6})$.

Abstract

Mean-Field Control (MFC) has recently been proven to be a scalable tool to approximately solve large-scale multi-agent reinforcement learning (MARL) problems. However, these studies are typically limited to unconstrained cumulative reward maximization framework. In this paper, we show that one can use the MFC approach to approximate the MARL problem even in the presence of constraints. Specifically, we prove that, an $N$-agent constrained MARL problem, with state, and action spaces of each individual agents being of sizes $|\mathcal{X}|$, and $|\mathcal{U}|$ respectively, can be approximated by an associated constrained MFC problem with an error, $e\triangleq \mathcal{O}\left([\sqrt{|\mathcal{X}|}+\sqrt{|\mathcal{U}|}]/\sqrt{N}\right)$. In a special case where the reward, cost, and state transition functions are independent of the action distribution of the population, we prove that the error can be improved to $e=\mathcal{O}(\sqrt{|\mathcal{X}|}/\sqrt{N})$. Also, we provide a Natural Policy Gradient based algorithm and prove that it can solve the constrained MARL problem within an error of $\mathcal{O}(e)$ with a sample complexity of $\mathcal{O}(e^{-6})$.