Scalable Multi-Agent Reinforcement Learning with General Utilities

TL;DR

This paper introduces a scalable distributed policy gradient algorithm for multi-agent reinforcement learning with general utilities, leveraging spatial correlation decay to ensure convergence without full observability.

Contribution

It presents the first scalable MARL algorithm for general utilities that converges efficiently without requiring full observability of all agents.

Findings

Algorithm converges to ε-stationarity with high probability.

Sample complexity is approximately O(ε^{-2}) with respect to the accuracy parameter.

Performance improves exponentially with increased communication radius.

Abstract

We study the scalable multi-agent reinforcement learning (MARL) with general utilities, defined as nonlinear functions of the team's long-term state-action occupancy measure. The objective is to find a localized policy that maximizes the average of the team's local utility functions without the full observability of each agent in the team. By exploiting the spatial correlation decay property of the network structure, we propose a scalable distributed policy gradient algorithm with shadow reward and localized policy that consists of three steps: (1) shadow reward estimation, (2) truncated shadow Q-function estimation, and (3) truncated policy gradient estimation and policy update. Our algorithm converges, with high probability, to $\epsilon$-stationarity with $\widetilde{\mathcal{O}}(\epsilon^{-2})$ samples up to some approximation error that decreases exponentially in the communication radius. This is the first result in the literature on multi-agent RL with general utilities that does not require the full observability.