A Distributed Primal-Dual Method for Constrained Multi-agent Reinforcement Learning with General Parameterization

TL;DR

This paper introduces a decentralized primal-dual algorithm for constrained multi-agent reinforcement learning, enabling agents to learn cooperatively without central coordination, with proven convergence and application to complex stochastic environments.

Contribution

It presents a novel distributed actor-critic based primal-dual method for CMARL with convergence guarantees and consensus on Lagrangian multipliers.

Findings

Algorithm converges to an equilibrium point with consensus among agents.

Proven sub-optimality bounds for the equilibrium compared to the exact solution.

Effective in a stochastic constrained Cournot game environment.

Abstract

This paper proposes a novel distributed approach for solving a cooperative Constrained Multi-agent Reinforcement Learning (CMARL) problem, where agents seek to minimize a global objective function subject to shared constraints. Unlike existing methods that rely on centralized training or coordination, our approach enables fully decentralized online learning, with each agent maintaining local estimates of both primal and dual variables. Specifically, we develop a distributed primal-dual algorithm based on actor-critic methods, leveraging local information to estimate Lagrangian multipliers. We establish consensus among the Lagrangian multipliers across agents and prove the convergence of our algorithm to an equilibrium point, analyzing the sub-optimality of this equilibrium compared to the exact solution of the unparameterized problem. Furthermore, we introduce a constrained cooperative Cournot game with stochastic dynamics as a test environment to evaluate the algorithm's performance in complex, real-world scenarios.