Feature-Based Q-Learning for Two-Player Stochastic Games

TL;DR

This paper introduces a feature-based Q-learning algorithm for two-player stochastic games that efficiently approximates Nash equilibria with sample complexity independent of the game's original dimensions.

Contribution

It proposes a novel two-player Q-learning method leveraging feature embeddings and develops an accelerated version with proven sample efficiency guarantees.

Findings

The basic algorithm finds an $oldsymbol{ ext{ extit{ extepsilon}}}$-optimal strategy with samples linear in features.

The accelerated algorithm achieves $oldsymbol{ ext{ extit{ extepsilon}}}$-optimality with $ ilde{oldsymbol{ ext{O}}}(K/( ext{ extit{ extepsilon}}^{2}(1-oldsymbol{ ext{ extit{ extgamma}}})^{4}))$ samples.

Sample, time, and space complexities are independent of the original game dimensions.

Abstract

Consider a two-player zero-sum stochastic game where the transition function can be embedded in a given feature space. We propose a two-player Q-learning algorithm for approximating the Nash equilibrium strategy via sampling. The algorithm is shown to find an $\epsilon$-optimal strategy using sample size linear to the number of features. To further improve its sample efficiency, we develop an accelerated algorithm by adopting techniques such as variance reduction, monotonicity preservation and two-sided strategy approximation. We prove that the algorithm is guaranteed to find an $\epsilon$-optimal strategy using no more than $\tilde{\mathcal{O}}(K/(\epsilon^{2}(1-\gamma)^{4}))$ samples with high probability, where $K$ is the number of features and $\gamma$ is a discount factor. The sample, time and space complexities of the algorithm are independent of original dimensions of the game.