Generalization in Mean Field Games by Learning Master Policies

TL;DR

This paper introduces the concept of Master policies in Mean Field Games, which can generalize across different initial distributions, enabling scalable multi-agent systems with a single policy that achieves Nash equilibrium.

Contribution

It proposes a novel method to learn Master policies using neural networks and reinforcement learning, demonstrating their effectiveness and generalization in large population MFGs.

Findings

Master policies provide Nash equilibrium for any initial distribution.

The proposed learning method effectively captures and generalizes policies.

Numerical examples show strong performance and generalization capabilities.

Abstract

Mean Field Games (MFGs) can potentially scale multi-agent systems to extremely large populations of agents. Yet, most of the literature assumes a single initial distribution for the agents, which limits the practical applications of MFGs. Machine Learning has the potential to solve a wider diversity of MFG problems thanks to generalizations capacities. We study how to leverage these generalization properties to learn policies enabling a typical agent to behave optimally against any population distribution. In reference to the Master equation in MFGs, we coin the term ``Master policies'' to describe them and we prove that a single Master policy provides a Nash equilibrium, whatever the initial distribution. We propose a method to learn such Master policies. Our approach relies on three ingredients: adding the current population distribution as part of the observation, approximating Master policies with neural networks, and training via Reinforcement Learning and Fictitious Play. We illustrate on numerical examples not only the efficiency of the learned Master policy but also its generalization capabilities beyond the distributions used for training.