Convergence Rates for Localized Actor-Critic in Networked Markov Potential Games

TL;DR

This paper introduces a scalable localized actor-critic algorithm for networked Markov potential games, providing finite-sample guarantees that do not depend on the total number of agents, thus addressing high-dimensional challenges.

Contribution

It proposes the first finite-sample bound for multi-agent competitive games that is independent of the number of agents, using local information and function approximation.

Findings

Achieves $ ilde{O}( ilde{ ext{epsilon}}^{-4})$ sample complexity.

Provides finite-sample guarantees up to localization and approximation errors.

First bound for multi-agent games independent of agent count.

Abstract

We introduce a class of networked Markov potential games in which agents are associated with nodes in a network. Each agent has its own local potential function, and the reward of each agent depends only on the states and actions of the agents within a neighborhood. In this context, we propose a localized actor-critic algorithm. The algorithm is scalable since each agent uses only local information and does not need access to the global state. Further, the algorithm overcomes the curse of dimensionality through the use of function approximation. Our main results provide finite-sample guarantees up to a localization error and a function approximation error. Specifically, we achieve an $\tilde{\mathcal{O}}(\tilde{\epsilon}^{-4})$ sample complexity measured by the averaged Nash regret. This is the first finite-sample bound for multi-agent competitive games that does not depend on the number of agents.