Solving Zero-Sum Convex Markov Games

TL;DR

This paper proves the first global convergence guarantees for independent policy gradient methods in two-player zero-sum convex Markov games, a broad class of multi-agent strategic models, using novel regularization techniques.

Contribution

It introduces a regularization approach that transforms the nonconvex min-max problem into a form with provable convergence guarantees for policy gradient algorithms.

Findings

Proves convergence of policy gradient methods to Nash equilibria in convex Markov games.

Develops a regularization technique stabilizing policy updates in complex multi-agent settings.

Provides the first global convergence guarantees for stochastic nested and alternating gradient methods.

Abstract

We contribute the first provable guarantees of global convergence to Nash equilibria (NE) in two-player zero-sum convex Markov games (cMGs) by using independent policy gradient methods. Convex Markov games, recently defined by Gemp et al. (2024), extend Markov decision processes to multi-agent settings with preferences that are convex over occupancy measures, offering a broad framework for modeling generic strategic interactions. However, even the fundamental min-max case of cMGs presents significant challenges, including inherent nonconvexity, the absence of Bellman consistency, and the complexity of the infinite horizon. We follow a two-step approach. First, leveraging properties of hidden-convex--hidden-concave functions, we show that a simple nonconvex regularization transforms the min-max optimization problem into a nonconvex-proximal Polyak-Lojasiewicz (NC-pPL) objective. Crucially, this regularization can stabilize the iterates of independent policy gradient methods and ultimately lead them to converge to equilibria. Second, building on this reduction, we address the general constrained min-max problems under NC-pPL and two-sided pPL conditions, providing the first global convergence guarantees for stochastic nested and alternating gradient descent-ascent methods, which we believe may be of independent interest.