Partial-Information Q-Learning for General Two-Player Stochastic Games

TL;DR

This paper introduces a partial-information Nash Q-learning algorithm for two-player stochastic games, proving its convergence to Nash equilibria without requiring players to know opponents' strategies, simplifying implementation.

Contribution

It presents the first convergence proof for partial-information Q-learning in general 2-player stochastic games, avoiding complex equilibrium computations at each step.

Findings

Partial-information Q-learning converges to Nash equilibria.

Performance comparable to full-information Q-learning and fictitious play.

Simplifies implementation by not requiring Nash equilibrium calculations each iteration.

Abstract

In this article we analyze a partial-information Nash Q-learning algorithm for a general 2-player stochastic game. Partial information refers to the setting where a player does not know the strategy or the actions taken by the opposing player. We prove convergence of this partially informed algorithm for general 2-player games with finitely many states and actions, and we confirm that the limiting strategy is in fact a full-information Nash equilibrium. In implementation, partial information offers simplicity because it avoids computation of Nash equilibria at every time step. In contrast, full-information Q-learning uses the Lemke-Howson algorithm to compute Nash equilibria at every time step, which can be an effective approach but requires several assumptions to prove convergence and may have runtime error if Lemke-Howson encounters degeneracy. In simulations, the partial information results we obtain are comparable to those for full-information Q-learning and fictitious play.