Mutation-Driven Follow the Regularized Leader for Last-Iterate Convergence in Zero-Sum Games

TL;DR

This paper introduces mutant FTRL, a variant of Follow the Regularized Leader, which guarantees last-iterate convergence to Nash equilibria in zero-sum games by incorporating mutation, outperforming existing methods in convergence speed and feedback scenarios.

Contribution

We propose M-FTRL, a novel mutation-based algorithm that ensures last-iterate convergence in zero-sum games and demonstrates superior performance over existing FTRL variants.

Findings

M-FTRL guarantees convergence to stationary points approximating Nash equilibria.

M-FTRL converges faster than FTRL and optimistic FTRL under full-information feedback.

M-FTRL exhibits clear convergence even under bandit feedback.

Abstract

In this study, we consider a variant of the Follow the Regularized Leader (FTRL) dynamics in two-player zero-sum games. FTRL is guaranteed to converge to a Nash equilibrium when time-averaging the strategies, while a lot of variants suffer from the issue of limit cycling behavior, i.e., lack the last-iterate convergence guarantee. To this end, we propose mutant FTRL (M-FTRL), an algorithm that introduces mutation for the perturbation of action probabilities. We then investigate the continuous-time dynamics of M-FTRL and provide the strong convergence guarantees toward stationary points that approximate Nash equilibria under full-information feedback. Furthermore, our simulation demonstrates that M-FTRL can enjoy faster convergence rates than FTRL and optimistic FTRL under full-information feedback and surprisingly exhibits clear convergence under bandit feedback.