Distributed primal-dual algorithm for constrained multi-agent reinforcement learning under coupled policies

TL;DR

This paper introduces a distributed primal-dual algorithm for constrained multi-agent reinforcement learning with coupled policies, ensuring agents optimize objectives while respecting safety constraints through local estimates and limited communication.

Contribution

It proposes a novel distributed primal-dual framework for CMARL with coupled policies, incorporating local estimates and neighborhood communication to enhance security and scalability.

Findings

Achieves $oldsymbol{ extit{ ext{epsilon}}}$-first-order stationarity with high probability.

Provides convergence bounds with approximation error depending on coupling and truncation distances.

Demonstrates effectiveness through simulations in GridWorld environment.

Abstract

In this work, we investigate constrained multi-agent reinforcement learning (CMARL), where agents collaboratively maximize the sum of their local objectives while satisfying individual safety constraints. We propose a framework where agents adopt coupled policies that depend on both local states and parameters, as well as those of their $\kappa_p$-hop neighbors, with $\kappa_p>0$ denoting the coupling distance. A distributed primal-dual algorithm is further developed under this framework, wherein each agent has access only to state-action pairs within its $2\kappa_p$-hop neighborhood and to reward information within its $\kappa + 2\kappa_p$-hop neighborhood, with $\kappa > 0$ representing the truncation distance. Moreover, agents are not permitted to directly share their true policy parameters or Lagrange multipliers. Instead, each agent constructs and maintains local estimates of these variables for other agents and employs such estimates to execute its policy. Additionally, these estimates are further updated and exchanged exclusively through an independent, time-varying networks, which enhances the overall system security. We establish that, with high probability, our algorithm can achieve an $\epsilon$-first-order stationary convergence with an approximation error of $\mathcal{O}(\gamma^{\frac{\kappa+1}{\kappa_{p}}})$ for discount factor $\gamma\in(0,1)$. Finally, simulations in GridWorld environment are conducted to demonstrate the effectiveness of the proposed algorithm.