Effects of noise on convergent game learning dynamics

TL;DR

This paper analyzes how small but finite noise influences the convergence and fluctuations of the lagging anchor reinforcement learning algorithm in iterated games, revealing noise-induced oscillations and potential payoff advantages.

Contribution

It provides an analytical framework for understanding stochastic effects on lagging anchor dynamics, including noise-driven quasicycles and asymmetric learning benefits.

Findings

Noise causes quasicycles in learning dynamics.

Asymmetric players can gain payoff advantages due to stochastic oscillations.

Fluctuation statistics can be accurately computed analytically.

Abstract

We study stochastic effects on the lagging anchor dynamics, a reinforcement learning algorithm used to learn successful strategies in iterated games, which is known to converge to Nash points in the absence of noise. The dynamics is stochastic when players only have limited information about their opponents' strategic propensities. The effects of this noise are studied analytically in the case where it is small but finite, and we show that the statistics and correlation properties of fluctuations can be computed to a high accuracy. We find that the system can exhibit quasicycles, driven by intrinsic noise. If players are asymmetric and use different parameters for their learning, a net payoff advantage can be achieved due to these stochastic oscillations around the deterministic equilibrium.