Dynamics of Boltzmann Q-Learning in Two-Player Two-Action Games

TL;DR

This paper analyzes how Boltzmann Q-learning dynamics behave in two-player, two-action games, revealing convergence to rest points that differ from Nash equilibria and how exploration noise influences these outcomes.

Contribution

It provides a comprehensive characterization of rest points in Boltzmann Q-learning and explores how exploration rates affect the stability and number of these points.

Findings

Strategies converge to rest points different from Nash equilibria.

Rest point structure is sensitive to exploration noise.

Multiple NE can lead to drastic changes in learning dynamics at critical exploration rates.

Abstract

We consider the dynamics of Q-learning in two-player two-action games with a Boltzmann exploration mechanism. For any non-zero exploration rate the dynamics is dissipative, which guarantees that agent strategies converge to rest points that are generally different from the game's Nash Equlibria (NE). We provide a comprehensive characterization of the rest point structure for different games, and examine the sensitivity of this structure with respect to the noise due to exploration. Our results indicate that for a class of games with multiple NE the asymptotic behavior of learning dynamics can undergo drastic changes at critical exploration rates. Furthermore, we demonstrate that for certain games with a single NE, it is possible to have additional rest points (not corresponding to any NE) that persist for a finite range of the exploration rates and disappear when the exploration rates of both players tend to zero.