On the Convergence of Model Free Learning in Mean Field Games

TL;DR

This paper proves the convergence of model-free learning algorithms in mean field games, showing they can reach approximate Nash equilibria in large population multi-agent systems, supported by theoretical analysis and numerical experiments.

Contribution

It provides the first convergence proof for model-free learning algorithms towards non-stationary mean field game equilibria under classical assumptions.

Findings

Convergence of model-free algorithms to approximate Nash equilibria.

Quantification of errors accumulated during learning iterations.

Numerical validation with deep reinforcement learning in continuous action spaces.

Abstract

Learning by experience in Multi-Agent Systems (MAS) is a difficult and exciting task, due to the lack of stationarity of the environment, whose dynamics evolves as the population learns. In order to design scalable algorithms for systems with a large population of interacting agents (e.g. swarms), this paper focuses on Mean Field MAS, where the number of agents is asymptotically infinite. Recently, a very active burgeoning field studies the effects of diverse reinforcement learning algorithms for agents with no prior information on a stationary Mean Field Game (MFG) and learn their policy through repeated experience. We adopt a high perspective on this problem and analyze in full generality the convergence of a fictitious iterative scheme using any single agent learning algorithm at each step. We quantify the quality of the computed approximate Nash equilibrium, in terms of the accumulated errors arising at each learning iteration step. Notably, we show for the first time convergence of model free learning algorithms towards non-stationary MFG equilibria, relying only on classical assumptions on the MFG dynamics. We illustrate our theoretical results with a numerical experiment in a continuous action-space environment, where the approximate best response of the iterative fictitious play scheme is computed with a deep RL algorithm.

Equations

$$x_{t+1}=x_t+b(x_t,a_t,{\mu }_t)+{ϵ}_{t+1},$$

$$J(x_0,\pi ,\mu ):=\mathbb{E}\left[\sum _{t=0}^{T-1}{\gamma }^{t}r(x_t^{\pi },{\mu }_t,a_t)\right],$$

$$\begin{array}{rl}\mathbb{J}(\pi ,\mu )& :={\mathbb{E}}_{x_0\sim {\mu }_0}[J(x_0,\pi ,\mu )],\end{array}$$

$${\ell }_n:=\mathbb{J}({\pi }^{\ast ,(n+1)},{\bar{\mu }}^{(n)})-\mathbb{J}({\hat{\pi }}^{(n+1)},{\bar{\mu }}^{(n)})\ge 0,$$

$$J(x_0,{\bar{\pi }}^{(n)},{\bar{\mu }}^{(n)}):=\frac{1}{n}\sum _{k=1}^{n}J(x_0,{\hat{\pi }}^{(k)},{\bar{\mu }}^{(n)}),n\in \mathbb{N}.$$

$$e_n$$

$${\mu }_0\left(\left\{x_0;J(x_0,{\pi }_{ϵ,\delta }^{\ast },{\mu }_{ϵ,\delta }^{\ast })\ge J(x_0,{\pi }^{\prime },{\mu }_{ϵ,\delta }^{\ast })-ϵ,\forall {\pi }^{\prime }\right\}\right)$$

$$\phi (x_0):=J(x_0,{\pi }^{\ast ,(n+1)},{\bar{\mu }}^{(n)})-J(x_0,{\bar{\pi }}^{(n)},{\bar{\mu }}^{(n)})\ge 0,$$

$${\mu }_0\left(\left\{x_0\in \mathcal{X}:\phi (x_0)\ge ϵ\right\}\right)$$

$$\le \frac{{\mathbb{E}}_{x_0\sim {\mu }_0}[\phi (x_0)]}{ϵ}=\frac{e_n}{ϵ},$$

$${\mu }_0\left(\left\{x_0;J(x_0,{\bar{\pi }}^{(n)},{\bar{\mu }}^{(n)})\ge J(x_0,{\pi }^{\ast ,(n+1)},{\bar{\mu }}^{(n)})-ϵ\right\}\right)$$

$$r(x_t^{\pi },{\mu }_t,a_t)\mapsto \tilde{r}(x_t^{\pi },a_t)+\bar{r}(x_t^{\pi },{\mu }_t),$$

$$\frac{1}{C}{I}_d\le D_{aa}\tilde{r}(x,.)\le C{I}_d,$$

$$\Vert \tilde{r}(.,.){\Vert }_{\infty }+\Vert \bar{r}(.,.){\Vert }_{\infty }\le C,$$

$${\int }_{\mathcal{X}}\left[\bar{r}(.,m_1)-\bar{r}(.,m_2)\right]d[m_1-m_2]<0.$$

$$d_1({\bar{\mu }}^{(n)},{\bar{\mu }}^{(n+1)})\le \frac{C}{n},n\in \mathbb{N},\text{mbox}forsomeC>0,$$

$$e_n$$

$${\hat{e}}_n:=\mathbb{J}({\hat{\pi }}^{(n+1)},{\bar{\mu }}^{(n)})-\mathbb{J}({\bar{\pi }}^{(n)},{\bar{\mu }}^{(n)})\ge 0,$$

$${\hat{e}}_{n+1}-\frac{n}{n+1}{\hat{e}}_n$$

$${\hat{e}}_{n+1}-\frac{n}{n+1}{\hat{e}}_n\le {\ell }_n+\frac{C}{n}{d}_1({\hat{\mu }}^{(n+2)},{\hat{\mu }}^{(n+1)}).$$

$$J(x_0,\pi ,\mu ):=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(x_t^{\pi },{\mu }_t,a_t)\right],$$

$$r(x_t,{\mu }_t,a_t)=\tilde{r}(x_t)-\frac{1}{2}|a_t{|}^2-\log ({\mu }_t),$$

$$a^{\ast }:x\mapsto \pi \cos (2\pi x)\text{mbox}and{\mu }^{\ast }:x\mapsto \frac{e^{2\sin (2\pi x)}}{{\int }_{\mathbf{T}}{e}^{2\sin (2\pi y)}dy},$$

$$d_1(\mu ,{\mu }^{\prime })=\underset{h\in Li{p}_1({\mathcal{X}}^T,\mathbb{R})}{\sup }\sum _{t\in \mathbb{T}}{\int }_{\mathcal{X}}h(x)d({\mu }_t-{\mu }_t^{\prime })(x),$$

$${\bar{\mu }}^{(n+1)}-{\bar{\mu }}^{(n)}$$

$$=\frac{1}{n+1}\left[{\hat{\mu }}^{(n+1)}-{\bar{\mu }}^{(n)}\right].$$

$$\left|{\int }_{{\mathcal{X}}^T}h(\mathbf{x})d({\bar{\mu }}^{(n+1)}-{\bar{\mu }}^{(n)})(\mathbf{x})\right|$$

$$=\frac{1}{n+1}\left|{\int }_{{\mathcal{X}}^T}(h(\mathbf{x})-h({\mathbf{x}}_0))d({\hat{\mu }}^{(n)}-{\bar{\mu }}^{(n)})(\mathbf{x})\right|$$

$$\le \frac{1}{n+1}{\int }_{{\mathcal{X}}^T}\Vert \mathbf{x}-{\mathbf{x}}_0\Vert \left[d{\hat{\mu }}^{(n)}(\mathbf{x})+d{\bar{\mu }}^{(n)}(\mathbf{x})\right]$$

$$\le \frac{C}{n+1},$$

Full text

1 Introduction

In Multi-agent systems (MAS), several autonomous robots or agents interact and cooperate, compete or coordinate in order to complete their task. The difficult nature of the task at hand combined with the large number of possible situations imply that the agents have to learn by experience. In comparison to the single-agent case, the derivation of efficient learning algorithms in this context is difficult due to the lack of stationarity of the environment, whose dynamics evolves as the population learns (?). This gives rise to research topics lying at the intersection of game theory and reinforcement learning. Nevertheless, in typical examples, the number of interacting agents can be very large (e.g., swarm systems) and defies the scalability properties of most learning algorithms. For anonymous identical agents, a key simplification in game theory is the introduction of the asymptotic limit where the number of agents is infinite, leading to the modeling intuition behind the theory of Mean Field Games (MFG). This calls for an analysis of model free learning scheme for MAS in terms of MFG.

MFG were introduced by ? (?; ?) and ? (?) in order to model the dynamic equilibrium between a large number of anonymous identical agents in interactions. Such systems encompass the modeling of numerous applications such as traffic jam dynamics, swarm systems, financial market equilibrium, crowd evacuation, smart grid control, web advertising auction, vaccination dynamics, rumor spreading on social media, among others. In a sequential game theory setting, each player needs to take into account his impact on the strategies of the other players. Studying games with an infinite number of players is easier from this point of view, as the impact of one single player on the others can be neglected. Hereby, the asymptotic limit with infinite population size considered in MFG becomes highly relevant. A solution to a dynamic MFG is determined via the optimal policy of a representative agent in response to the flow of the entire population. A mean field (MF) Nash equilibrium arises when the distribution of the best response policies over the population generates the exact same population flow. In most cases, a MF Nash equilibrium provides an approximate Nash equilibrium for an analogous game with a finite number of players (?; ?; ?).

In the abundant literature on MFG, most papers consider planning problems with fully informed agents about the game operation scheme, the reward function and the MF population dynamics. Only a few contributions focus on learning problems in MFG, see e.g. (?; ?; ?; ?) for model based approaches. Very recently, a rapidly growing literature intends to approximate the solution of stationary MFG in the realistic setting where agents with no prior information on the game learn their best response policy through repeated experience. These contributions restrict to a stationary setting and focus on specific Reinforcement Learning (RL) algorithms: Q-learning (?; ?), fictitious play (?) or policy gradient methods (?), and sometimes rely on hardly verifiable assumptions.

In this paper, we take a step back and adopt a general high perspective on the convergence of model free learning algorithms in possibly non-stationary MFG and emphasize their potential for MAS with a large number of agents. Our approach investigates how any single-agent learning algorithm can perform in an MFG setting, in order to learn a (possibly approximate) Nash equilibrium, via repeated experiences and without any prior knowledge. Namely, we quantify precisely how the convergence of model free iterative learning algorithms reduces to the error analysis of each learning iteration step, in analogy with how the convergence of RL algorithms reduces to the aggregation of repeated supervised learning approximation errors (?; ?). For this purpose, our approach relies on a model free Fictitious Play (FP) iterative learning scheme for repeated games, where each agent calibrates its belief to the empirical frequency of the previously observed population flows. The FP approach is very natural when agents are trying to learn how to play a game by experience, while interacting with others. Before a new round of experience, they need to anticipate the behavior of the other players, and FP ergodic averaging is nicely designed for this purpose. This algorithm is typically useful for building from experience collaboration a cooperation patterns in a MAS using a decentralized learning scheme. In our framework of interest, all agents are identical (as usual in MFG), and we consistently suppose that they use the same learning scheme. Whenever the agents can compute their exact best response to any population flow, FP is proved to reach asymptotically a Nash equilibrium in some (but not all (?)) classes of games, such as first order monotone MFG (?). However, in a realistic setting, the agents are not able to compute their exact best response and can only attain an approximate version of it. This induces at each iteration a learning approximation error, which propagates through the FP learning scheme.

The main contribution of this paper is theoretical, as we provide a rigorous study of the error propagation in Approximate FP algorithms for MFGs, using an innovative line of proof in comparison to the standard two time scale approximation convergence results (?; ?). Our convergence results are derived under easily verifiable assumptions on possibly non-stationary MFG dynamics and cost, which are highly classical in the MFG literature (namely $1^{st}$ order monotone MFGs). This allows discussing the convergence to a (possibly approximate) MF Nash equilibrium, when using any standard single-agent learning algorithm as an inner step embedded in a FP iterative scheme. Especially, our theoretical framework encompasses the convergence of RL algorithms to MFG equilibria in non stationary settings, which, as far as we know, is new in the literature. We illustrate our theoretical results on an authorative MFG numerical experiment on crowd congestion, where the approximate best response of the iterative FP scheme is computed with a deep RL algorithm. This provides for the first time a model free learning example on MFG in a continuous state-action environment.

2 Background

Mean Field Games.

MFGs were introduced by  ? (?; ?) and by ? (?) and correspond to the asymptotic limit of a differential game, where the number of agents is infinite. Since all agents are assumed to be identical and indistinguishable, individual interactions are irrelevant in the limit and only the distribution of states matters (see  (?) for a complete overview). Most of the MFG literature is displayed in continuous time, but we choose to present our analysis in a discrete time setting in order to alleviate the presentation and emphasize the fruitful connections with the learning literature.

Finding a Mean Field Nash equilibrium boils down to identifying the equilibrium distribution dynamics of the population as well as the best response (or optimal policy) of a representative agent to this population mean field flow. Since the number of players is infinite, each agent has an infinitesimal influence on the population distribution. Yet, since all agents are rational, at equilibrium the state distribution generated by the optimal policy must coincide with the population distribution.

Notations. Let $\mathcal{X}$ and $\mathcal{A}$ be compact convex subsets of $\mathbb{R}^{\ell}$ and $\mathbb{R}^{d}$ respectively, which represent the state and action spaces common to every agent. Let $T>0$ be a time horizon and let $\mathbb{T}$ denote the time sequence $\{0,1,\dots,T\}$. We denote by $\mathcal{P}(\mathcal{X})$ the set of probability measures on $\mathcal{X}$ and by $M_{T}=\mathcal{P}(\mathcal{X})^{T}$ the set of all possible flows of population state distributions $\mu=(\mu_{0},\mu_{1},...,\mu_{T})$. The initial distribution of the population is an atomless measure on $\mathcal{X}$ denoted by $\mu_{0}$. For $\mu\in M_{T}$, $\mu_{t}$ represents the distribution at time $t$ of the state occupation of the entire population.

State dynamics $\&$ Mean field population flow. At any time $t\in\mathbb{T}$, each agent belongs to a state $x_{t}\in\mathcal{X}$ and picks an action $a_{t}\in\mathcal{A}$. For a sequence of actions $a:=(a_{t})_{t\in\mathbb{T}}$, the dynamics of $x$ is governed by a Markov Decision Process (MDP) with (possibly non-stationary) transition density $p_{t}(.|x_{t},a_{t},\mu_{t})$ parameterized by the mean field flow $\mu\in M_{T}$ of the population. This indexation transcribes the interactions with the other agents, through their state distribution $\mu_{t}$. Typically, the dynamics of $x$ is described by an equation of the form

[TABLE]

where $b:\mathcal{X}\times\mathcal{A}\times\mathcal{P}(\mathcal{X})\to\mathcal{X}$ is a drift function and $(\epsilon_{t})_{t\geq 1}$ is a dynamic source of noise. We stress that the mean field term $\mu_{t}$ represents the whole population distribution and not just the average state, as e.g. in (?).

We denote by $\Pi_{T}$ the set of policies (or controls) $\pi:\mathbb{T}\times\mathcal{X}\to\mathcal{A}$ which are feedback in the state: at time $t$, an agent using policy $\pi$ while in state $x_{t}$ plays the action $a_{t}=\pi(t,x_{t})$. The process $x$ controlled by $\pi\in\Pi_{T}$ is denoted $x^{\pi}$.

Agent’s reward scheme. An infinitesimal agent starting at time [math] in state $x_{0}\in\mathcal{X}$ chooses a policy $\pi\in\Pi_{T}$ in order to maximize the following discounted expected sum of rewards:

[TABLE]

while interacting with the population MF flow $\mu\in M_{T}$. At time $t$, the agent’s rewards are impacted by $\mu_{t}$, which represents the aggregate state distribution of all the other agents (i.e. of the whole population). Since the agents are anonymous, only the MF distribution flow $\mu$ of the states matters.

As $\mu_{0}$ denotes the state distribution at time [math], the average reward for a representative agent is given by

[TABLE]

when this agent uses policy $\pi\in\Pi_{T}$, while the mean field population flow is $\mu$.

Definition 1** (Best response).**

A policy $\pi^{*}$ maximizing $\mathbb{J}(.,\mu)$ is called a best response of the representative agent to the MF population dynamic flow $\mu$.

MF Nash Equilibrium. While interacting, the agents may or may not reach a Nash equilibrium, whose definition, based on the previous best response policy characterization, reads as follows:

Definition 2** (Mean Field Nash equilibrium).**

A pair $(\pi^{*}_{t},\mu^{*}_{t})_{t\in\mathbb{T}}$ consisting of a policy and a MF population distribution flow is called MF Nash equilibrium if it satisfies

• •

Agent rationality:* $\pi^{*}$ is a best response to $\mu^{*}$;*

• •

Population consistency:* for all $t\in\mathbb{T}$, $\mu^{*}_{t}$ is the distribution of $x^{*}_{t}$, starting with distribution $\mu_{0}$ and controlled by policy $\pi^{*}$.*

Namely, if the mean field population flow is $\mu^{*}$, the policy $\pi^{*}$ is optimal, and if all the agents play according to $\pi^{*}$, the induced mean field population flow coincides with $\mu^{*}$. Hereby, $(\pi^{*},\mu^{*})$ identifies to an MF Nash equilibrium.

Observe that reaching an MF Nash equilibrium requires the computation of the exact best response policy, which can be difficult in practice. We are concerned with the design of an iterative learning scheme, where the available best response is partially accurate and typically approximated by RL through repeated experiences. For example, this realistic situation arises when agents are repeatedly optimizing their daily driving trajectories, without any prior information on the traffic jam dynamics.

3 Fictitious Play Algorithms for MFG

Fictitious play (?) is an iterative learning scheme for repeated games, where each agent calibrates its belief to the empirical frequency of previously observed strategies of other agents, and plays optimally according to its beliefs. This constitutes its best response. Even in simple two-player games, the convergence of FP to a Nash equilibrium is not guaranteed (?). However, the convergence of FP has recently been proved for some classes of MFG (?; ?).

Yet, in most cases, agents do not have access to the exact best response policy $\pi^{*}$ but use an approximate version of it instead, in the spirit of (?; ?). At iteration $n$, the agent has only access to an approximate version $\hat{\pi}^{n+1}$ of the best response $\pi^{*,n+1}$ to the anticipated mean field flow $\bar{\mu}^{(n)}$, defined precisely in Algorithm 1.

At iteration step $n$, the learning scheme induces an average additional error $\ell_{n}$ defined as

[TABLE]

for $n\in\mathbb{N}$. Observe that $\ell_{n}$ identifies to the expected loss over the entire population at step $n$, when replacing the the exact best response $\pi^{*,(n+1)}$ by the approximate policy $\hat{\pi}^{(n+1)}$.

In Section 4 below, we quantify the propagation of approximating errors $\ell_{n}$ and clarify the convergence properties of Algorithm 1 for any type of learning procedure at each intermediate step. The specific setting where the approximate optimal policy is computed using single-agent RL algorithms is discussed in Sec 4.3.

Approximate Nash equilibrium

At each step $n$, we denote by $\bar{\pi}^{(n)}$ the representative agent belief on the aggregate population policy, defined as an equally randomized version of all previous approximate best responses $(\hat{\pi}^{(k)})_{k=1,\ldots,n}$: for each $t\in\mathbb{T}$ and $x\in\mathcal{X}$, $\bar{\pi}^{(n)}(t,x)\in\mathcal{P}(\mathcal{A})$ is the probability distribution on the set of actions $\mathcal{A}$ according to which the player picks uniformly at random an element of $\{\hat{\pi}^{(1)}(t,x),\dots,\hat{\pi}^{(n)}(t,x)\}$.

With a slight abuse of notation, we write

[TABLE]

and modify the definition of $\mathbb{J}$ in (3) accordingly. Observe for later use that, by construction, $\bar{\mu}^{(n)}$ defined in Algorithm 1 coincides with the population MF flow induced by the policy $\bar{\pi}^{(n)}$. In order to assess the quality of $(\bar{\mu}^{(n)},\bar{\pi}^{(n)})$ as an (approximate) MF Nash equilibrium, we introduce, for $n\in\mathbb{N}$,

[TABLE]

The exploitability $e_{n}$ quantifies at iteration $n$ the expected gain for a typical agent, when shifting its belief $\bar{\pi}^{(n)}$ to the exact best response $\pi^{*,(n+1)}$, while interacting with the MF population flow $\bar{\mu}^{(n)}$. After $n$ iterations in Algorithm 1, $e_{n}$ is a quantitative measure of the quality of $(\bar{\mu}^{(n)},\bar{\pi}^{(n)})$ as an MF Nash equilibrium. For the sake of clarification, let us introduce a more precise weaker notion of MF Nash equilibrium, inspired by (?).

Definition 3** (Approximate MF Nash equilibrium).**

For $\epsilon>0$ and $\delta\in(0,1)$, a pair $(\pi^{*}_{\epsilon,\delta},\mu^{*}_{\epsilon,\delta})$ consisting of a policy and a population distribution flow is called an $(\epsilon,\delta)-$MF Nash equilibrium if

[TABLE]

is at least $1-\delta$, and $\mu^{*}_{\epsilon,\delta}$ coincides with the MF distribution flow starting from $\mu_{0}$, when every agent uses policy $\pi^{*}_{\epsilon,\delta}$.

An $(\epsilon,\delta)-$MF Nash equilibrium identifies to a weak equilibrium which reveals $\epsilon$-optimal for at least a fraction $(1-\delta)$ of the population. We are now in position to clarify how the exploitability $e_{n}$ quantifies the quality of $(\bar{\mu}^{(n)},\bar{\pi}^{(n)})$ as an MF Nash equilibrium.

Theorem 4**.**

If $0\leq e_{n}\leq\epsilon^{2}$ for some $n\in\mathbb{N}$, then $(\bar{\mu}^{(n)},\bar{\pi}^{(n)})$ is an $(\epsilon,\epsilon)$-MF Nash equilibrium in the sense of Definition 3. If $e_{n}$ goes to 0 as $n\to+\infty$, any accumulation point of $\bar{\mu}^{(n)}$ is a MF Nash equilibrium

Proof.

Fix $n\in\mathbb{N}$ and assume $0\leq e_{n}\leq\epsilon^{2}.$ Let us introduce

[TABLE]

as $\pi^{*,(n+1)}$ is the best response to the MF flow $\bar{\mu}^{(n)}$.

Using Markov’s inequality and the bound on $e_{n}$ we obtain

[TABLE]

which is smaller than $\epsilon$. Collecting the terms and using the definition of $\varphi$, we deduce that

[TABLE]

is at least $1-\epsilon$, so that $(\bar{\pi}^{(n)},\bar{\mu}^{(n)})$ is an $(\epsilon,\epsilon)$-MF Nash equilibrium.

The second part of the theorem follows directly. ∎

4 Error propagation & Nash equilibrium approximation for first order MFG

Since the exploitability $e_{n}$ identifies to a relevant quality measure of Algorithm 1 after $n$ iterations, we now evaluate how the individual learning errors $(\ell_{k})_{0\leq k\leq n}$ aggregate over $e_{n}$. For the sake of simplicity, we focus our discussion on $1^{\text{st}}$ order MFG, i.e. without source of noise in the dynamics. This allows us to build our reasoning on the analysis of (?, Chapter 3) and to avoid a limitative restriction to second order games with a potential structure, for which similar results should hold in that setting as well, see ? (?).

4.1 First order mean field game

The state $(x_{t})_{t}$ evolves in $\mathbb{R}^{d}$ with dynamics (1), where we take $b(x,a,\mu)=a$, and $(\epsilon_{t})_{t}=0$. In other words, each agent controls exactly its state variation between two time steps and does not endure any noise. While interacting with a MF flow $\mu\in M_{T}$, each agent intends to maximize the classical reward scheme given by (2) with a running reward at time $t$ of the form:

[TABLE]

where the extra $\bar{r}$ captures the impact of the other agents’ positions. In Sec. 5, we provide in particular a congestion example where $\bar{r}$ models an appeal for non-crowded regions. This type of conditions translates into the so-called Lasry-Lions monotonicity condition ? (?; ?) which ensures uniqueness of MF Nash equilibrium. More precisely, existence and uniqueness of solution to the $1^{\text{st}}$ order MFG of interest hold under the following classical set of assumptions.

Assumption 1**.**

For some constant $C$, the reward functions $\tilde{r}$ and $\bar{r}$ satisfy:

• •

For any $x\in\mathcal{X}$, the map $\tilde{r}(x,.)$ is twice differentiable and

[TABLE]

• •

The function $\bar{r}$ is continuous on $:\mathcal{X}\times\mathcal{P}(\mathcal{X})$ and $\bar{r}(.,m)$ is $\mathcal{C}^{1}$ on $\mathcal{X}$ ,

• •

We have

[TABLE]

• •

The Lasry-Lions monotonicity condition holds: for all $m_{1},m_{2}\in\mathcal{P}(\mathcal{X})$,

[TABLE]

4.2 Error propagation in the Fictitious Play algorithm

We now investigate how the learning error $(\ell_{n})_{n}$ propagates through FP for any learning algorithm, while Sec 4.3 focuses on the specific case where the best response is approximated via RL.

The key ingredient of FP iterative learning schemes is the quick stabilization of the sequence of beliefs $(\bar{\mu}^{(n)})_{n}$.

Lemma 5**.**

Under Assumption 1, the FP MF flow $\bar{\mu}^{(n)}$ satisfies:

[TABLE]

where $d_{1}$ is the Wasserstein distance.

The proof follows from a straightforward adaptation of  (?, Lemma 3.3.2) to our setting.

As the sequence of beliefs $\bar{\mu}_{n}$ stabilizes, the impact of recent learning errors $(\ell_{n})$ reduces and we are in position to quantify the global error $e_{n}$ of the algorithm after $n$ iteration steps. This is the main result of the paper, whose proof interestingly differs from the more classical two-time scale approximation argument (?).

Theorem 6**.**

Under Assumption 1, the Nash equilibrium quality $(e_{n})_{n}$ satisfies both estimates: for all $n\in\mathbb{N}$

[TABLE]

for some constants $C_{1}$ and $C_{2}$.

Sketch of Proof.

Our argumentation builds up on the exact FP analysis of (?, Theorem 3.3.1), which hereby extends to the approximate best response setting.

Let us introduce the approximate exploitability

[TABLE]

so that $e_{n}=\ell_{n}+\hat{e}_{n}$, for $n\in\mathbb{N}$. In order to control the exploitability $e_{n}$, we focus our analysis on $\hat{e}_{n}$. Denoting $J_{n}:x\in\mathcal{X}^{T}\mapsto J(x_{0},(x_{t+1}-x_{t})_{t=0,\dots,T-1},\bar{\mu}^{(n)})$, we get:

[TABLE]

where the last equality follows from the definition of $\bar{\mu}^{(n)}$.

The monotonicity of the reward in Assumption 1 implies

[TABLE]

Besides, Assumption 1 together with the compactness of $\mathcal{X}$ and (?, Lemma 3.5.2) and Lemma 5 imply that $J_{n+1}-J_{n}$ is $C/n$-Lipschitz, leading to

[TABLE]

As $\pi^{*,(n+1)}$ is the best response to the mean field flow $\bar{\mu}^{(n)}$, recalling the definition of $\ell_{n}$ in (4), we deduce

[TABLE]

Together with estimate $e_{n}=\ell_{n}+\hat{e}_{n}$ and (?, Lemma 3.3.1), we derive (8) and conclude the proof. ∎

Bound (7) indicates a nice averaging aggregation of the learning errors $(\ell_{n})_{n}$, but requires a strong additional control on the Wasserstein distance between the MF flows generated by both approximate and exact best responses. Such estimate is readily available for the numerical approximation of convex stochastic control problems (?) but less classical in the RL literature, as discussed in Sec 4.3. When such an estimate is not available, Bound (8) provides a slower $n\ell_{n}$ convergence rate, up to a weak $d_{1}$-regularity of the approximate best response in terms of the mean field flow $\bar{\mu}^{(n)}$, recall Lemma 5. Such estimate is highly classical in the setting of convex stochastic control problems with Lipschitz rewards (?; ?).

At finite distance, the following corollary sums up these properties in terms of MF Nash equilibrium.

Corollary 7**.**

Under Assumption 1, if ever \frac{1}{n}\sum_{i=1}^{n}\big{(}\ell_{i}+C_{1}d_{1}(\mu^{*,(i)},\hat{\mu}^{(i)}))\big{)} or \frac{1}{n}\sum_{i=1}^{n}\big{(}(i+1)\ell_{i}+C_{2}d_{1}(\hat{\mu}^{(i+1)},\hat{\mu}^{(i)}))\big{)} is bounded by $\mathbb{\epsilon}^{2}/2$, $(\bar{\mu}^{(n)},\bar{\pi}^{(n)})$ is an $(\mathbb{\epsilon},\mathbb{\epsilon})$-MF Nash equilibrium, for $n$ large enough.

In a similar fashion, we can conclude on the general asymptotic convergence of Algorithm 1 to the unique MF-Nash equilibrium, before discussing the specific implications for RL best response approximation schemes.

Corollary 8**.**

Under Assumption 1, the approximate FP algorithm converges to the unique MF Nash equilibrium whenever one of the following two conditions holds:

The approximate best response update procedure $\bar{\mu}^{(n)}\mapsto\hat{\mu}^{(n+1)}$ is continuous in $d_{1}$, and $n\ell_{n}\to 0$, as $n\to\infty$ ; 2. 2.

The learning and policy approximation errors $\ell_{n}$ and $(d_{1}(\mu^{*,(n)},\hat{\mu}^{(n)}))_{n}$ converge to [math].

The convergence of the sequence $(\bar{\mu}^{(n)})_{n}$ follows from the tightness and pre-compactness property of this collection of measures with respect to the Wasserstein distance, see e.g. Remark 3.5.3 in (?).

4.3 Discussion on the convergence for Best Response RL approximation

The result in Theorem 6 is general and relies on standard assumptions of MFGs. It also relies on a good enough control of the approximation error on the best response at each iteration. Here, we discuss to what extent existing theoretical results for RL algorithms allow satisfying this assumption.

As stated in Corollary 8, in order for the approximate FP to converge to the exact MF Nash equilibrium, the approximate best response should converge quickly enough to the best one, depending on the number of iterations. From an RL perspective, this would require being able to compute the approximate optimal policy to an arbitrary precision, with high probability. As far as we know, such a result is possible only when an exact representation of any value function is possible, that is, in the tabular setting which imposes finite state and action spaces. Notably, convergence and rate of convergence of Q-learning-like algorithms have been studied in the literature, see e.g. (?; ?; ?; ?). For example, the speedy Q-learning algorithm requires $\mathcal{O}(\ln(K)/(\epsilon^{2}(1-\gamma)^{4}))$ steps to learn an $\epsilon$-optimal state-action value function with high probability, with $K$ the number of state-action couples. According to Corollary 8, if the error is in $\mathcal{O}(n^{-\alpha})$ with $\alpha>1$ (and if we have continuity in $d_{1}$), then the scheme converges to the Nash equilibrium. This suggests using $\mathcal{O}(\ln(K)n^{2\alpha}/(1-\gamma)^{4})$ steps for the RL agent at iteration $n$. Yet, this kind of results does not provide guarantees on the continuity in $d_{1}$.

According to Corollary 7, bounding the learning errors and the distance between two iterates of the distribution is sufficient to reach an approximate Nash equilibrium. As approximate FP can be seen as repeated RL problems, RL (or approximate dynamic programming) can be seen as repeated supervised learning problems, and the propagation of errors from supervised to RL is a well studied field, see e.g. (?; ?). Basically, if the supervised learning steps are bounded by some $\mathcal{O}(\epsilon)$, then the learning error of the RL algorithm is bounded by $\mathcal{O}(C\epsilon/(1-\gamma)^{2})$, where $C$ is a so -called concentrability coefficient, measuring the mismatch between some measures. In principle, we could then propagate the learning error of the supervised learning part up to the FP error, through the RL error. However, these results do not provide any guarantees on the proximity between the estimated optimal policy and the actual one (which would be a sufficient condition for the proximity between population distributions); it only provides a guarantee on the distance between their respective returns. This is due to the fact that in RL, the optimal value function is unique, but not the optimal policy. A perspective would be to consider regularized MDPs (?), where the optimal policy is unique (and greediness is Lipschitz). Yet, this would come at the cost of a bias in the Nash equilibria. The approach in (?) somehow builds partially on this idea in their specific learning scheme.

5 Numerical illustration

As an illustration, we consider a stylized authoritative MFG model with congestion in the spirit of ? (?). This application should be seen as a proof of concept showing that the method described above can be applied beyond the framework used for our theoretical results. We compute a model free approximation of the MFG solution combining Algorithm 1 with Deep Deterministic Policy Gradient (DDPG) (?). As far as we know, this is the first numerical illustration of model free deep RL Algorithm for MFG with continuous states and actions. Our numerical results also demonstrate the empirical convergence of the Fictitious RL scheme in a larger setting, even when the MFG is of not first order type.

As usual in RL, instead of (2) we consider the problem in infinite horizon with the following discounted reward:

[TABLE]

when an infinitesimal player interacts with the population MF flow $\mu=(\mu_{0},\mu_{1},\mu_{2},\dots)$. The goal is to learn the policy which is optimal in the long run, i.e., when the behavior of the population becomes stationary.

Environment

Each agent has a position $x$ located on the torus $\mathbf{T}=[0,1]$ with periodic boundary conditions (for simplicity of explicit solution), whose dynamics is governed by $x_{t+1}=x_{t}+a_{t}\Delta_{t}+\sqrt{\Delta_{t}}\epsilon_{t},\quad t=0,1,\dots,$ where $\Delta_{t}$ is the time step of the continuous time process. It receives the per-step reward

[TABLE]

where the last term motivates agents to avoid congestion, i.e. the proximity to a region with a large population density. In the continuous time setting with no discounting, a direct PDE argument provides the ergodic solution $(a^{*},\mu^{*})$ in closed form (?)

[TABLE]

when the geographic reward is of the form $\tilde{r}(x_{t})=2\pi^{2}\sin(2\pi x_{t})-2\pi^{2}\cos(2\pi x_{t})^{2}+2\sin(2\pi x_{t})$. This closed form solution offers a nice benchmark for our experiments and allows to measure the errors made by our algorithm.

Implemented Algorithm

Model free FP for MFGs takes a somehow similar approach as ? (?) in the sense that we estimate the best response using a model free RL algorithm (namely DDPG). However we do not maintain those best responses as in (?) but rather learn the population MF flow of the distribution of the representative agents. The best response approximation through DDPG and the estimation of the population MF are left in the Algorithm. We ran $30000$ trajectories of DDPG with a trajectory length of $300$. The noise used for exploration is a centered normal noise with variance $0.02$ and we used Adam optimizers with $0.001$ starting learning rate and $\tau=0.01$. At each iteration of FP, we added $N_{\textrm{trajectories FP}}=3000$ trajectories of length $1000$ to the replay buffer. Finally, we estimated the density using $100$ classes and doing $30000$ steps of Adam (with $0.001$ initial learning rate).

Results.

Figure 1 presents the learned equilibrium computed for $\gamma=0.95$, $N=90$ and uniform initial distribution, as well as the continuous time closed form ergodic solution for $\gamma=1$, see (10). We emphasize that the variation in $\gamma$ together with the discrete/continuous time difference setting implies that the theoretical solutions to both problems are close but do not exactly coincide. We keep this benchmark since no ergodic closed form solution is available for $\gamma\neq 1$. As observed on Figure 1, both ergodic explicit and learned distributions and controls are close. As expected, the density of players is larger around the point of maximum of the reward but the distribution is not highly concentrated due to the logarithmic penalty encoding aversion for congested regions. More precisely, Figure 1 indicates that the $L_{2}$ errors between the distributions and the controls decrease with the number $n$ of iterations. The convergence of control distributions echoes to the discussion on error propagation in Section 4.2. This clearly illustrates the numerical convergence of the Deep RL FP mean field algorithm.

6 Related work

The related literature is as follows. Recently, model free RL algorithms for solving MFGs were analyzed in the following papers: ? (?) and ? (?) study $Q$-learning, ? (?) consider FP but contains several inaccuracies, as already pointed out in ? (?), which focuses on policy gradient methods. However, their studies are restricted to a stationary setting and focus on particular RL algorithms. Their convergence results hold under assumptions that are often hard to verify in practice. Although not focusing on an MFG, ? (?) uses the idea of MF approximation by considering interactions through the empirical mean action. Numerical illustrations provided in all these papers are in a finite state-action setting, while we present a numerical example in a continuous state-action setting. On a different note, ? (?) studies the link between MFG and inverse RL. Some authors also study “learning” algorithms which use the full knowledge of the model (and hence are not model-free): ? (?) studied a MF oscillator game while ? (?) proposed a decentralized deep FP learning architecture for large MARL, whose convergence holds on linear quadratic MFG examples with explicit solution and small maturity.

7 Conclusion and future research

In comparison to the existing literature focusing on specific RL algorithms for MFGs, we took a step back and offer a general perspective on the error propagation in iterative scheme for MFG, using any learning algorithm. We presented a rigorous convergence analysis of model free FP learning algorithm for MF Agent systems, encompassing cases where the best response is approximated using any single agent learning algorithm as well as non-stationary settings. We showed how the convergence of model free iterative FP algorithm reduces to the error analysis of each learning iteration step, as the convergence of RL algorithm reduces to the aggregation of repeated supervised learning approximation errors (?; ?). Our theoretical setting covers for the first time the consideration of non-stationary MFG and relies on reasonable and verifiable assumptions on the MFG of interest. The convergence is illustrated for the first time by numerical experiments in a continuous state-action setting, based on deep RL algorithm. Our analysis motivates and properly justifies the use of asymptotic Mean Field approximation for the study of learning by experience schemes in Multi-Agent systems, with a large number of agents.

For RL approximation schemes, our analysis suggests a much faster convergence rate, whenever the best response approximation quality can be controlled in Wasserstein distance. This kind of estimate is classical in the numerical approximation of stochastic control literature but currently not available in the RL literature. The derivation of such estimate deserves to be addressed in future research papers. Finally, we focused on convergence properties for a centralized Multi-Agent learning algorithm, paving the way for addressing such property for a more relevant decentralized one.

Appendix

This Appendix regroups the technical proofs related to the error propagation bounds on the Approximate Fictitious Play algorithm detailed in Theorem 6. Many arguments reported here are inspired by the results presented in (?) for the exact fictitious play algorithm.

We follow the notations of Section 4. In particular, we recall that $\mathcal{X}$ denotes the set of accessible states before time $T$. Since we take $b(x,a,\mu)=a$ and the set $\mathcal{A}$ is compact, $\mathcal{X}^{T}=\mathcal{X}\times\dots\times\mathcal{X}$ is compact too and in particular it is a bounded subset of $\mathbb{R}^{T\times d}$.

In order to measure the proximity between MF population flows, we denote by $d_{1}$, the $1-$ Wasserstein distance defined (using Kantorovitch-Rubinstein duality) as: for all $\mu,\mu^{\prime}\in M_{T}$,

[TABLE]

where $Lip_{1}(\mathcal{X}^{T},\mathbb{R})$ is the set of $1$-Lipschitz continuous function from $\mathcal{X}^{T}$ to $\mathbb{R}$.

A. Stability of the FP mean field flow $(\bar{\mu}^{(n)})_{n}$

Let us first provide the proof of Lemma 5 which ensures the closeness in $d_{1}$ of two consecutive elements of the Mean field flow learning sequence $\bar{\mu}^{(n)}$. Let first recall from the definition of $\bar{\mu}^{(n)}$ that we have: for all $n\in\mathbb{N}$,

[TABLE]

Proof of Lemma 5.

Let $h\in Lip_{1}(\mathcal{X}^{T},\mathbb{R})$. We recall that $\mathcal{X}^{T}$ is bounded and pick $\mathbf{x}_{0}\in\mathcal{X}^{T}$. Then, using (11) together with the definition of $\bar{\mu}^{(n)}$, we compute

[TABLE]

since $\mathcal{X}^{T}$ is bounded. This result being valid for any $h\in Lip_{1}(\mathcal{X}^{T},\mathbb{R})$, we obtain

[TABLE]

∎

B. Propagation error estimates

This section is dedicated to the rigorous derivation of the bounds (8) and (7) presented in Theorem 4.

We first recall the following useful result, see e.g. (?, Lemma 3.3.1).

Lemma 9**.**

Let $(\varphi_{n})_{n}$ and $(\lambda_{n})_{n}$ be two sequences of real numbers such that

[TABLE]

Then, we have the estimate:

[TABLE]

For ease of notation, we introduce $J_{n}:=J(.,\bar{\mu}^{(n)})$ and $J_{n+1}:=J(.,\bar{\mu}^{(n+1)})$ (which are functions defined over $\mathcal{X}^{T}$), for all $n\geq 0$. More precisely, for $x\in\mathcal{X}^{T}$,

[TABLE]

Observe that this definition is accurate by the definition of the first order MFG setting presented in Section 4.1 and because there is a bijection between process trajectory and the combination of initial position and policy.

Proof of estimate (8) in Theorem 6.

We adapt the arguments in the proof of (?, Theorem 3.3.1) to our setting with approximate best responses.

Let us introduce the approximate learning error $\hat{e}_{n}$ defined by: for $n\in\mathbb{N}$,

[TABLE]

so that $e_{n}=\ell_{n}+\hat{e}_{n}$. In order to control $e_{n}$, we will focus our analysis on $\hat{e}_{n}$ and compute

[TABLE]

where the last equality follows from (11).

Thanks to Assumption 1, the monotonicity of the reward function implies directly

[TABLE]

By definition of $J_{n}$ together with the first order MFG dynamics, we have the expression

[TABLE]

Moreover, using Assumption 1 and the compactness of $\mathcal{X}$, we deduce as in (?, Lemma 3.5.2), the existence of a constant $C$ such that for all $\mathbf{x},\mathbf{x}^{\prime}\in\mathcal{X}^{T}$,

[TABLE]

This property of the reward function together with Lemma 5 indicate that $J_{n+1}-J_{n}$ is $C/n$-Lipschitz, so that

[TABLE]

where, in the last inequality, we used the fact that $\pi^{*,(n+1)}$ is the best response with respect to $\bar{\mu}^{(n)}$ and hence

[TABLE]

Indeed, by optimality of $\pi^{*,(n+1)}$, we have

[TABLE]

By definition of the learning error $\ell_{n}$ in (4), we deduce

[TABLE]

By Lemma 9 applied to

[TABLE]

we derive the estimate

[TABLE]

Combining this estimate with the relation $e_{n}=\ell_{n}+\hat{e}_{n}$ provides (8). ∎

We are now in position to turn to the proof of the remaining estimate (7).

Proof of estimate (7) in Theorem 6.

Combining the relation $e_{n}=\ell_{n}+\hat{e}_{n}$ together with estimate (12), we compute

[TABLE]

Hence, the Lipschitz property (13) together with Lemma 5 implies

[TABLE]

where we also used that $\mu^{*,(n+1)}$ is the mean field flow induced by the best response to the population distribution $\bar{\mu}^{(n)}$ so that

[TABLE]

Finally, Lemma 5 together with the continuity of the best response in the first order MFG (which stems from Assumption 1, see e.g. (?, Remark 3.5.3)) ensure that $d_{1}(\mu^{*,(n+2)},\mu^{*,(n+1)})$ converges to zero as $n$ goes to infinity. Finally, applying Lemma 9 to

[TABLE]

directly concludes the proof. ∎

C. Algorithms

For sake of completeness, we detail here the pseudo code for important algorithms of the paper: first exact fictitious play, then Deep Deterministic Policy Gradient.

D. Approximation of the density

The density estimation is done through classification. We divide a state dataset $X$ of size $N$ into classes representing a partitioning of the space $\left([\frac{i}{N_{\textrm{classes}}},\frac{i+1}{N_{\textrm{classes}}}]\right)_{i\in\{0,...,N_{\textrm{classes}}-1\}}$. Then we use a function $f_{\theta}\in\{0\}\rightarrow\Delta_{N_{\textrm{classes}}}$ to estimate the density of $x\in X$ being in this interval $[\frac{i}{N_{\textrm{classes}}},\frac{i+1}{N_{\textrm{classes}}}]$ by minimizing the cross entropy loss $L(\theta)=\frac{1}{N}\sum\limits_{i=1}^{N}\sum\limits_{j=1}^{N_{\textrm{classes}}}\mathds{1}_{x_{i}\in[\frac{j}{N_{\textrm{classes}}},\frac{j+1}{N_{\textrm{classes}}}]}\log(f_{\theta}^{j})$.

E. Additional numerical results

We provide in this Section additional numerical results on the illustrative example presented in Section 5. Figure 2 presents the ergodic continuous time theoretical optimal policy together with the numerically estimated optimal one for the discrete time model, with parameters detailed in Section 5.