On the Power of Perturbation under Sampling in Solving Extensive-Form Games

TL;DR

This paper explores how perturbation techniques, specifically in the form of Perturbed FTRL algorithms, can stabilize learning and improve convergence in solving imperfect-information extensive-form games with sampling noise, introducing two variants and analyzing their performance.

Contribution

It introduces a unified framework for Perturbed FTRL algorithms, including PFTRL-KL and PFTRL-RKL, and demonstrates their effectiveness and variance-reduction benefits in game-solving scenarios.

Findings

PFTRL-RKL outperforms PFTRL-KL in asymmetric Leduc poker.

Variance reduction in RKL improves last-iterate convergence.

Perturbation stabilizes learning under sampling noise.

Abstract

We investigate how perturbation does and does not improve the Follow-the-Regularized-Leader (FTRL) algorithm in solving imperfect-information extensive-form games under sampling, where payoffs are estimated from sampled trajectories. While optimistic algorithms are effective under full feedback, they often become unstable in the presence of sampling noise. Payoff perturbation offers a promising alternative for stabilizing learning and achieving \textit{last-iterate convergence}. We present a unified framework for \textit{Perturbed FTRL} algorithms and study two variants: PFTRL-KL (standard KL divergence) and PFTRL-RKL (Reverse KL divergence), the latter featuring an estimator with both unbiasedness and conditional zero variance. While PFTRL-KL generally achieves equivalent or better performance across benchmark games, PFTRL-RKL consistently outperforms it in Leduc poker, whose structure is more asymmetric than the other games in a sense. Given the modest advantage of PFTRL-RKL, we design the second experiment to isolate the effect of conditional zero variance, showing that the variance-reduction property of RKL improve last-iterate performance.