Shaping Large Population Agent Behaviors Through Entropy-Regularized Mean-Field Games

TL;DR

This paper introduces entropy-regularized mean-field games with finite states and actions, providing algorithms for finding unique equilibria and demonstrating how a reference policy can control large population behaviors.

Contribution

It establishes the regularity conditions enabled by entropy regularization, develops fixed-point algorithms for equilibrium computation, and shows how reference policies influence population behavior.

Findings

Entropy regularization ensures regularity conditions in finite MFGs.

Fixed-point algorithms can find the unique mean-field equilibrium.

Reference policies effectively control large population behaviors.

Abstract

Mean-field games (MFG) were introduced to efficiently analyze approximate Nash equilibria in large population settings. In this work, we consider entropy-regularized mean-field games with a finite state-action space in a discrete time setting. We show that entropy regularization provides the necessary regularity conditions, that are lacking in the standard finite mean field games. Such regularity conditions enable us to design fixed-point iteration algorithms to find the unique mean-field equilibrium (MFE). Furthermore, the reference policy used in the regularization provides an extra parameter, through which one can control the behavior of the population. We first consider a stochastic game with a large population of $N$ homogeneous agents. We establish conditions for the existence of a Nash equilibrium in the limiting case as $N$ tends to infinity, and we demonstrate that the Nash equilibrium for the infinite population case is also an $\epsilon$-Nash equilibrium for the $N$-agent system, where the sub-optimality $\epsilon$ is of order $\mathcal{O}\big(1/\sqrt{N}\big)$. Finally, we verify the theoretical guarantees through a resource allocation example and demonstrate the efficacy of using a reference policy to control the behavior of a large population.