Local and adaptive mirror descents in extensive-form games

TL;DR

This paper introduces an adaptive online mirror descent algorithm for learning near-optimal strategies in zero-sum imperfect information games, reducing variance and improving convergence rates through fixed sampling and local updates.

Contribution

It proposes a novel adaptive OMD approach with local updates and decreasing learning rates, achieving near-optimal convergence in extensive-form games with fixed sampling.

Findings

Guarantees a convergence rate of .5 with high probability.

Achieves near-optimal dependence on game parameters with optimal learning rates and sampling policies.

Generalizes OMD stabilization to include time-varying regularization.

Abstract

We study how to learn $\epsilon$-optimal strategies in zero-sum imperfect information games (IIG) with trajectory feedback. In this setting, players update their policies sequentially based on their observations over a fixed number of episodes, denoted by $T$. Existing procedures suffer from high variance due to the use of importance sampling over sequences of actions (Steinberger et al., 2020; McAleer et al., 2022). To reduce this variance, we consider a fixed sampling approach, where players still update their policies over time, but with observations obtained through a given fixed sampling policy. Our approach is based on an adaptive Online Mirror Descent (OMD) algorithm that applies OMD locally to each information set, using individually decreasing learning rates and a regularized loss. We show that this approach guarantees a convergence rate of $\tilde{\mathcal{O}}(T^{-1/2})$ with high probability and has a near-optimal dependence on the game parameters when applied with the best theoretical choices of learning rates and sampling policies. To achieve these results, we generalize the notion of OMD stabilization, allowing for time-varying regularization with convex increments.