(Doubly) Exponential Lower Bounds for Follow the Regularized Leader in Potential Games

TL;DR

This paper proves that the Follow the Regularized Leader algorithm can take exponential or doubly exponential time to converge to Nash equilibria in potential games, revealing fundamental limitations of this widely used method.

Contribution

It establishes the first exponential lower bounds for FTRL in potential games and shows doubly exponential convergence time for fictitious play, a special case.

Findings

FTRL can require exponential time to reach Nash equilibrium in two-player potential games.

Fictitious play can take doubly exponential time to converge in multi-player potential games.

An exponential upper bound matches the lower bound up to factors in the exponent.

Abstract

Follow the regularized leader FTRL is the premier algorithm for online optimization. However, despite decades of research on its convergence in constrained optimization -- and potential games in particular -- its behavior remained hitherto poorly understood. In this paper, we establish that FTRL can take exponential time to converge to a Nash equilibrium in two-player potential games for any (permutation-invariant) regularizer and potentially vanishing learning rate. By known equivalences, this translates to an exponential lower bound for certain mirror descent counterparts, most notably multiplicative weights update. On the positive side, we establish the potential property for FTRL and obtain an exponential upper bound $\exp(O_{\epsilon}(1/\epsilon^2))$ for any no-regret dynamics executed in a lazy, alternating fashion, matching our lower bound up to factors in the exponent. Finally, in multi-player potential games, we show that fictitious play -- the extreme version of FTRL -- can take doubly exponential time to reach a Nash equilibrium. This constitutes an exponentially stronger lower bound for the foundational learning algorithm in games.