Reinforcement Learning for Mean Field Games with Strategic Complementarities

TL;DR

This paper introduces a new equilibrium concept called T-MFE for mean field games with strategic complementarities, along with algorithms for computing and learning it, including online methods, supported by empirical evaluations.

Contribution

It proposes T-MFE, a refinement of equilibrium for MFGs with strategic complementarities, and develops algorithms for its computation and learning, including online approaches.

Findings

Proposed a new equilibrium concept T-MFE for MFGs.

Developed algorithms with known and unknown models for learning T-MFE.

Empirically validated the algorithms with real-world motivated examples.

Abstract

Mean Field Games (MFG) are the class of games with a very large number of agents and the standard equilibrium concept is a Mean Field Equilibrium (MFE). Algorithms for learning MFE in dynamic MFGs are unknown in general. Our focus is on an important subclass that possess a monotonicity property called Strategic Complementarities (MFG-SC). We introduce a natural refinement to the equilibrium concept that we call Trembling-Hand-Perfect MFE (T-MFE), which allows agents to employ a measure of randomization while accounting for the impact of such randomization on their payoffs. We propose a simple algorithm for computing T-MFE under a known model. We also introduce a model-free and a model-based approach to learning T-MFE and provide sample complexities of both algorithms. We also develop a fully online learning scheme that obviates the need for a simulator. Finally, we empirically evaluate the performance of the proposed algorithms via examples motivated by real-world applications.