Markov Games with Decoupled Dynamics: Price of Anarchy and Sample Complexity

TL;DR

This paper analyzes Markov games with decoupled dynamics, establishing bounds on the price of anarchy, introducing a distributed learning algorithm for potential games, and validating results through a dynamic covering game.

Contribution

It extends smoothness concepts to Markov games with decoupled dynamics, introduces the MA-SPI algorithm, and provides sample complexity analysis.

Findings

Bounded the price of anarchy using smoothness in Markov games.

Developed the MA-SPI algorithm with convergence guarantees.

Validated theoretical results with a dynamic covering game.

Abstract

This paper studies the finite-time horizon Markov games where the agents' dynamics are decoupled but the rewards can possibly be coupled across agents. The policy class is restricted to local policies where agents make decisions using their local state. We first introduce the notion of smooth Markov games which extends the smoothness argument for normal form games to our setting, and leverage the smoothness property to bound the price of anarchy of the Markov game. For a specific type of Markov game called the Markov potential game, we also develop a distributed learning algorithm, multi-agent soft policy iteration (MA-SPI), which provably converges to a Nash equilibrium. Sample complexity of the algorithm is also provided. Lastly, our results are validated using a dynamic covering game.