The Dynamics of Q-learning in Population Games: a Physics-Inspired Continuity Equation Model

TL;DR

This paper introduces a physics-inspired continuity equation model to accurately describe the dynamics of Q-learning in large-scale population games, improving understanding and parameter tuning in multi-agent systems.

Contribution

A novel continuity equation-based model for Q-learning dynamics in population games, addressing limitations of previous replicator models and applicable to various game settings.

Findings

Model accurately describes Q-learning dynamics across different initial conditions.

Applicable to various exploration mechanisms and game configurations.

Provides insights for parameter tuning and system behavior analysis.

Abstract

Although learning has found wide application in multi-agent systems, its effects on the temporal evolution of a system are far from understood. This paper focuses on the dynamics of Q-learning in large-scale multi-agent systems modeled as population games. We revisit the replicator equation model for Q-learning dynamics and observe that this model is inappropriate for our concerned setting. Motivated by this, we develop a new formal model, which bears a formal connection with the continuity equation in physics. We show that our model always accurately describes the Q-learning dynamics in population games across different initial settings of MASs and game configurations. We also show that our model can be applied to different exploration mechanisms, describe the mean dynamics, and be extended to Q-learning in 2-player and n-player games. Last but not least, we show that our model can provide insights into algorithm parameters and facilitate parameter tuning.