Model-Based Reinforcement Learning for Offline Zero-Sum Markov Games

TL;DR

This paper introduces a simple, model-based offline reinforcement learning algorithm for two-player zero-sum Markov games that provably finds approximate Nash equilibria with near-optimal sample complexity, improving upon prior methods.

Contribution

The paper proposes VI-LCB-Game, a pessimistic model-based algorithm with Bernstein-style confidence bounds, achieving minimax optimal sample complexity for offline zero-sum Markov games.

Findings

Achieves near-optimal sample complexity for offline Nash equilibrium learning.

Strengthens previous bounds by a factor of {A,B} and is minimax optimal.

Algorithmic simplicity without variance reduction or sample splitting.

Abstract

This paper makes progress towards learning Nash equilibria in two-player zero-sum Markov games from offline data. Specifically, consider a $\gamma$-discounted infinite-horizon Markov game with $S$ states, where the max-player has $A$ actions and the min-player has $B$ actions. We propose a pessimistic model-based algorithm with Bernstein-style lower confidence bounds -- called VI-LCB-Game -- that provably finds an $\varepsilon$-approximate Nash equilibrium with a sample complexity no larger than $\frac{C_{\mathsf{clipped}}^{\star}S(A+B)}{(1-\gamma)^{3}\varepsilon^{2}}$ (up to some log factor). Here, $C_{\mathsf{clipped}}^{\star}$ is some unilateral clipped concentrability coefficient that reflects the coverage and distribution shift of the available data (vis-\`a-vis the target data), and the target accuracy $\varepsilon$ can be any value within $\big(0,\frac{1}{1-\gamma}\big]$. Our sample complexity bound strengthens prior art by a factor of $\min\{A,B\}$, achieving minimax optimality for the entire $\varepsilon$-range. An appealing feature of our result lies in algorithmic simplicity, which reveals the unnecessity of variance reduction and sample splitting in achieving sample optimality.