Roping in Uncertainty: Robustness and Regularization in Markov Games

TL;DR

This paper establishes a connection between robust Markov games with uncertainty and regularized Markov games, providing algorithms and complexity results for computing robust Nash equilibria under various uncertainty structures.

Contribution

It introduces an equivalence between robust and regularized Markov games, offers a planning algorithm, and identifies conditions for polynomial-time solutions in certain uncertainty settings.

Findings

Equivalence between robust Nash equilibrium and regularized Markov game solutions.

PPAD-hardness of computing RNE in reward-uncertain two-player zero-sum matrix games.

Polynomial-time solvability under efficient player-decomposability for certain uncertainty sets.

Abstract

We study robust Markov games (RMG) with $s$-rectangular uncertainty. We show a general equivalence between computing a robust Nash equilibrium (RNE) of a $s$-rectangular RMG and computing a Nash equilibrium (NE) of an appropriately constructed regularized MG. The equivalence result yields a planning algorithm for solving $s$-rectangular RMGs, as well as provable robustness guarantees for policies computed using regularized methods. However, we show that even for just reward-uncertain two-player zero-sum matrix games, computing an RNE is PPAD-hard. Consequently, we derive a special uncertainty structure called efficient player-decomposability and show that RNE for two-player zero-sum RMG in this class can be provably solved in polynomial time. This class includes commonly used uncertainty sets such as $L_1$ and $L_\infty$ ball uncertainty sets.