The impact of uncertainty on regularized learning in games

TL;DR

This paper studies how randomness affects learning in games, showing that uncertainty pushes players towards pure strategies and alters the stability of equilibria, with implications for predicting outcomes under noisy conditions.

Contribution

It introduces a perturbed FTRL model analyzing the impact of uncertainty, revealing that only pure Nash equilibria are stable under stochastic dynamics.

Findings

Players reach near pure strategies in finite time under uncertainty.

Pure Nash equilibria are the only stable limits of the dynamics.

Randomness causes recurrent game dynamics to drift toward the boundary.

Abstract

In this paper, we investigate how randomness and uncertainty influence learning in games. Specifically, we examine a perturbed variant of the dynamics of "follow-the-regularized-leader" (FTRL), where the players' payoff observations and strategy updates are continually impacted by random shocks. Our findings reveal that, in a fairly precise sense, "uncertainty favors extremes": in any game, regardless of the noise level, every player's trajectory of play reaches an arbitrarily small neighborhood of a pure strategy in finite time (which we estimate). Moreover, even if the player does not ultimately settle at this strategy, they return arbitrarily close to some (possibly different) pure strategy infinitely often. This prompts the question of which sets of pure strategies emerge as robust predictions of learning under uncertainty. We show that (a) the only possible limits of the FTRL dynamics under uncertainty are pure Nash equilibria; and (b) a span of pure strategies is stable and attracting if and only if it is closed under better replies. Finally, we turn to games where the deterministic dynamics are recurrent - such as zero-sum games with interior equilibria - and we show that randomness disrupts this behavior, causing the stochastic dynamics to drift toward the boundary on average.