A Policy-Gradient Approach to Solving Imperfect-Information Games with Best-Iterate Convergence

TL;DR

This paper demonstrates that a policy gradient method can be used in two-player zero-sum imperfect-information games to provably converge to a Nash equilibrium, bridging a gap between reinforcement learning and game theory.

Contribution

It introduces a policy gradient approach with theoretical guarantees of convergence in imperfect-information extensive-form games, a novel result in the field.

Findings

Proves policy gradient convergence to Nash equilibrium in self-play

Establishes best-iterate convergence guarantees

Bridges reinforcement learning and game theory in imperfect-information settings

Abstract

Policy gradient methods have become a staple of any single-agent reinforcement learning toolbox, due to their combination of desirable properties: iterate convergence, efficient use of stochastic trajectory feedback, and theoretically-sound avoidance of importance sampling corrections. In multi-agent imperfect-information settings (extensive-form games), however, it is still unknown whether the same desiderata can be guaranteed while retaining theoretical guarantees. Instead, sound methods for extensive-form games rely on approximating \emph{counterfactual} values (as opposed to Q values), which are incompatible with policy gradient methodologies. In this paper, we investigate whether policy gradient can be safely used in two-player zero-sum imperfect-information extensive-form games (EFGs). We establish positive results, showing for the first time that a policy gradient method leads to provable best-iterate convergence to a regularized Nash equilibrium in self-play.