Markov $\alpha$-Potential Games

TL;DR

This paper introduces Markov $oldsymbol{ extalpha}$-potential games, a new framework for analyzing Markov games, establishing existence of potential functions, and proposing algorithms with regret bounds, applicable to congestion and team games.

Contribution

The paper develops the concept of Markov $oldsymbol{ extalpha}$-potential games, proves their properties, and introduces algorithms with theoretical guarantees for equilibrium approximation.

Findings

Any finite-state, finite-action Markov game is a Markov $oldsymbol{ extalpha}$-potential game.

Explicit bounds for $oldsymbol{ extalpha}$ are derived for specific game classes.

Projected gradient and sequential improvement algorithms achieve bounded Nash regret.

Abstract

We propose a new framework of Markov $\alpha$-potential games to study Markov games. We show that any Markov game with finite-state and finite-action is a Markov $\alpha$-potential game, and establish the existence of an associated $\alpha$-potential function. Any optimizer of an $\alpha$-potential function is shown to be an $\alpha$-stationary Nash equilibrium. We study two important classes of practically significant Markov games, Markov congestion games and the perturbed Markov team games, via the framework of Markov $\alpha$-potential games, with explicit characterization of an upper bound for $\alpha$ and its relation to game parameters. Additionally, we provide a semi-infinite linear programming based formulation to obtain an upper bound for $\alpha$ for any Markov game. Furthermore, we study two equilibrium approximation algorithms, namely the projected gradient-ascent algorithm and the sequential maximum improvement algorithm, along with their Nash regret analysis, and corroborate the results with numerical experiments.