Linear quadratic mean field games with a major player: The multi-scale approach

TL;DR

This paper investigates linear quadratic mean field games with a major player, using a multi-scale approach to analyze asymptotic solvability, derive Riccati equations, and interpret strategies as best responses in an infinite population.

Contribution

It introduces a re-scaling technique to reduce coupled equations, providing necessary and sufficient conditions for asymptotic solvability and linking strategies to mean field approximations.

Findings

Derived Riccati equations in lower dimensions for solvability

Established conditions for asymptotic solvability

Interpreted strategies as best responses in an infinite population

Abstract

This paper considers linear quadratic (LQ) mean field games with a major player and analyzes an asymptotic solvability problem. It starts with a large-scale system of coupled dynamic programming equations and applies a re-scaling technique introduced in Huang and Zhou (2018a, 2018b) to derive a set of Riccati equations in lower dimensions, the solvability of which determines the necessary and sufficient condition for asymptotic solvability. We next derive the mean field limit of the strategies and the value functions. Finally, we show that the two decentralized strategies can be interpreted as the best responses of a major player and a representative minor player embedded in an infinite population, which have the property of consistent mean field approximations.

Equations

$$\displaystyle dX_{0}(t)=\$$

$$+D_{0}d{W}_0(t),$$

$$\displaystyle dX_{i}(t)=\$$

$$+Dd{W}_i(t),1\le i\le N,t\ge 0,$$

$$\displaystyle J_{0}(u)=\$$

$$+E|X_0(T)-{\Gamma }_{0f}{X}^{(N)}(T)-{\eta }_{0f}{|}_{Q_{0f}}^2,$$

$$\displaystyle J_{i}(u)=\$$

$$+E|X_i(T)-{\Gamma }_{1f}{X}_0(T)-{\Gamma }_{2f}{X}^{(N)}(T)-{\eta }_f{|}_{Q_f}^2,$$

$$1\le i\le N.$$

$$\text{mathbold}x=(\text{mathbold}{x}_0^T,\text{mathbold}{x}_1^T,\cdots ,\text{mathbold}{x}_N^T{)}^T\in {\mathbb{R}}^{(N+1)n},$$

$$X(t)=\left[\begin{array}{c}{X}_0(t)\\ ⋮\\ X_N(t)\end{array}\right]\in {\mathbb{R}}^{(N+1)n},W(t)=\left[\begin{array}{c}{W}_0(t)\\ ⋮\\ W_N(t)\end{array}\right]\in {\mathbb{R}}^{(N+1)n_2},$$

$$\widehat{\text{mathbold}}A=\text{mbox}diag[A_0,A,\cdots ,A]+\left[\begin{array}{ccc}& 0,& 1_{1\times n}\otimes \frac{F_0}{N}\\ & 1_{n\times 1}\otimes G,& 1_{n\times n}\otimes \frac{F}{N}\end{array}\right],$$

$$\widehat{\text{mathbold}}D=\text{mbox}diag[D_0,D,\cdots ,D]\in {\mathbb{R}}^{(N+1)n\times (N+1)n_2},$$

$$\text{mathbold}{B}_0=e_1^{N+1}\otimes B_0\in {\mathbb{R}}^{(N+1)n\times n_1},$$

$$\text{mathbold}{B}_k=e_{k+1}^{N+1}\otimes B\in {\mathbb{R}}^{(N+1)n\times n_1},1\le k\le N.$$

$$\displaystyle dX(s)=\Big{(}\widehat{\mathbold}{A}X(s)+\sum_{k=0}^{N}\mathbold{B}_{k}u_{k}(s)\Big{)}dt+\widehat{\mathbold}{D}dW(s),$$

$$J_k({\hat{u}}_k,{\hat{u}}_{-k})\le J_k(u_k,{\hat{u}}_{-k}),$$

$$\displaystyle 0=\frac{\partial V_{0}}{\partial t}+\min_{u_{0}\in{\mathbb{R}}^{n_{1}}}\Big{(}\frac{\partial^{T}V_{0}}{\partial\mathbold{x}}\Big{(}\widehat{\mathbold{A}}\mathbold{x}+\sum_{k=0}^{N}{\mathbold{B}}_{k}u_{k}\Big{)}+u_{0}^{T}R_{0}u_{0}$$

$$\displaystyle\qquad+|\mathbold{x}_{0}-\Gamma_{0}\mathbold{x}^{(N)}-\eta_{0}|_{Q_{0}}^{2}+\frac{1}{2}\mbox{Tr}\big{(}{\widehat{\mathbold{D}}^{T}(V_{0})_{\mathbold{x}\mathbold{x}}\widehat{\mathbold{D}}}\big{)}\Big{)},$$

$$V_0(T,\text{mathbold}x)=|\text{mathbold}{x}_0-{\Gamma }_{0f}\text{mathbold}{x}^{(N)}-{\eta }_{0f}{|}_{Q_{0f}}^2,$$

$$\displaystyle 0=\frac{\partial V_{i}}{\partial t}+\min_{u_{i}\in{\mathbb{R}}^{n_{1}}}\Big{(}\frac{\partial^{T}V_{i}}{\partial\mathbold{x}}\Big{(}\widehat{\mathbold{A}}\mathbold{x}+\sum_{k=0}^{N}{\mathbold{B}}_{k}u_{k}\Big{)}+u_{i}^{T}Ru_{i}$$

$$\displaystyle\qquad+|\mathbold{x}_{i}-\Gamma_{1}\mathbold{x}_{0}-\Gamma_{2}\mathbold{x}^{(N)}-\eta|_{Q}^{2}+\frac{1}{2}\mbox{Tr}\big{(}{\widehat{\mathbold{D}}^{T}(V_{i})_{\mathbold{x}\mathbold{x}}\widehat{\mathbold{D}}}\big{)}\Big{)},$$

$$V_i(T,\text{mathbold}x)=|\text{mathbold}{x}_i-{\Gamma }_{1f}\text{mathbold}{x}_0-{\Gamma }_{2f}\text{mathbold}{x}^{(N)}-{\eta }_f{|}_{Q_f}^2,1\le i\le N,$$

$$u_0=-\frac{1}{2}{R}_0^{-1}{\text{mathbold}B}_0^T\frac{\partial V_0}{\partial \text{mathbold}x},$$

$$u_i=-\frac{1}{2}{R}^{-1}{\text{mathbold}B}_i^T\frac{\partial V_i}{\partial \text{mathbold}x},1\le i\le N.$$

$$0=$$

$$\displaystyle-\frac{1}{2}\sum_{k=1}^{N}{\mathbold{B}}_{k}R^{-1}{\mathbold{B}}_{k}^{T}\frac{\partial V_{k}}{\partial\mathbold{x}}\Big{)}+|\mathbold{x}_{0}-\Gamma_{0}\mathbold{x}^{(N)}-\eta_{0}|_{Q_{0}}^{2}$$

$$\displaystyle+\frac{1}{4}\frac{\partial^{T}V_{0}}{\partial\mathbold{x}}{\mathbold{B}}_{0}R_{0}^{-1}\mathbold{B}_{0}^{T}\frac{\partial V_{0}}{\partial\mathbold{x}}+\frac{1}{2}\mbox{Tr}\big{(}{{\widehat{\mathbold}{D}}^{T}(V_{0})_{\mathbold{x}\mathbold{x}}{\widehat{\mathbold}{D}}}\big{)},$$

$$0=$$

$$\displaystyle-\frac{1}{2}\sum_{k=1}^{N}{\mathbold{B}}_{k}R^{-1}{\mathbold{B}}_{k}^{T}\frac{\partial V_{k}}{\partial\mathbold{x}}\Big{)}+|\mathbold{x}_{i}-\Gamma_{1}\mathbold{x}_{0}-\Gamma_{2}\mathbold{x}^{(N)}-\eta|_{Q}^{2}$$

$$+\frac{1}{4}\frac{{\partial }^T{V}_i}{\partial \text{mathbold}x}{\text{mathbold}B}_i{R}^{-1}\text{mathbold}{B}_i^T\frac{\partial V_i}{\partial \text{mathbold}x}$$

$$\displaystyle+\frac{1}{2}\mbox{Tr}\big{(}{{\widehat{\mathbold}{D}}^{T}(V_{i})_{\mathbold{x}\mathbold{x}}{\widehat{\mathbold}{D}}}\big{)},\qquad 1\leq i\leq N.$$

$$V_j(t,\text{mathbold}x)=\text{mathbold}{x}^T{\text{mathbold}P}_j(t)\text{mathbold}x+2{\text{mathbold}S}_j^T(t)\text{mathbold}x+\text{mathbold}{r}_j(t),$$

$$\frac{\partial V_j}{\partial \text{mathbold}x}=2{\text{mathbold}P}_j(t)\text{mathbold}x+2{\text{mathbold}S}_j(t),\frac{{\partial }^2{V}_j}{\partial \text{mathbold}{x}^2}=2\text{mathbold}{P}_j(t).$$

$$\text{mathbold}{K}_0=[I_n,0,\cdots ,0]-\frac{1}{N}[0,{\Gamma }_0,\cdots ,{\Gamma }_0],$$

$$\text{mathbold}{Q}_0=\text{mathbold}{K}_0^T{Q}_0\text{mathbold}{K}_0,$$

$$\text{mathbold}{K}_i=[0,0,\cdots ,I_n,0,\cdots ,0]-[{\Gamma }_1,0,\cdots ,0]$$

$$-\frac{1}{N}[0,{\Gamma }_2,\cdots ,{\Gamma }_2],$$

$$\text{mathbold}{Q}_i=\text{mathbold}{K}_i^{T}Q\text{mathbold}{K}_i,$$

$$|\text{mathbold}{x}_0-{\Gamma }_0\text{mathbold}{x}^{(N)}-{\eta }_0{|}_{Q_0}^2=\text{mathbold}{x}^T\text{mathbold}{Q}_0\text{mathbold}x-2\text{mathbold}{x}^T\text{mathbold}{K}_0^T{Q}_0{\eta }_0$$

$$+{\eta }_0^T{Q}_0{\eta }_0,$$

$$|\text{mathbold}{x}_i-{\Gamma }_1\text{mathbold}{x}_0-{\Gamma }_2\text{mathbold}{x}^{(N)}-\eta {|}_Q^2=\text{mathbold}{x}^T\text{mathbold}{Q}_i\text{mathbold}x-2\text{mathbold}{x}^T\text{mathbold}{K}_i^{T}Q\eta$$

$$+{\eta }^{T}Q\eta ,1\le i\le N.$$

$$\displaystyle\begin{cases}\dot{\mathbold{P}}_{0}(t)=-\big{(}{\mathbold{P}}_{0}\widehat{\mathbold{A}}+\widehat{\mathbold{A}}^{T}\mathbb{\mathbold{P}}_{0}\big{)}+{\mathbold{P}}_{0}{\mathbold{B}}_{0}R_{0}^{-1}{\mathbold{B}}_{0}^{T}{\mathbold{P}}_{0}\\ \qquad\quad+\Big{(}{\mathbold{P}}_{0}\sum_{k=1}^{N}{\mathbold{B}}_{k}R^{-1}{\mathbold{B}}_{k}^{T}{\mathbold{P}}_{k}\\ \qquad\quad+\sum_{k=1}^{N}{\mathbold{P}}_{k}{\mathbold{B}}_{k}R^{-1}{\mathbold{B}}_{k}^{T}{\mathbold{P}}_{0}\Big{)}-{\mathbold{Q}}_{0},\\ {\mathbold{P}}_{0}(T)={\mathbold{Q}}_{0f},\end{cases}$$

$$\{\begin{array}{l}{\dot{\text{mathbold}S}}_0(t)=-{\widehat{\text{mathbold}A}}^T{\text{mathbold}S}_0+{\text{mathbold}P}_0{\text{mathbold}B}_0{R}_0^{-1}{\text{mathbold}B}_0^T{\text{mathbold}S}_0\\ +{\sum }_{k=1}^N{\text{mathbold}P}_k{\text{mathbold}B}_k{R}^{-1}{\text{mathbold}B}_k^T{\text{mathbold}S}_0\\ +{\text{mathbold}P}_0{\sum }_{k=1}^N{\text{mathbold}B}_k{R}^{-1}{\text{mathbold}B}_k^T{\text{mathbold}S}_k+\text{mathbold}{K}_0^T{Q}_0{\eta }_0,\\ {\text{mathbold}S}_0(T)=-\text{mathbold}{K}_{0f}^T{Q}_{0f}{\eta }_{0f},\end{array}$$

Full text

††thanks: This paper was not presented at any IFAC meeting. This work was supported by the National Science Foundation of China (No.11601489), Startup Research Fund of Zhengzhou University (No.129-51090091), Outstanding Young Talent Research Fund of Zhengzhou University (No.129-32210453), Natural Sciences and Engineering Research Council (NSERC) of Canada. Submitted to Automatica, Jan 2019; revised Aug 2019. This version contains a more detailed Sec. 5 than the revised journal submission. Corresponding author: M. Huang.

,

1 Introduction

Mean field game theory has undergone a phenomenal growth. It provides a powerful methodology for handling complexity in noncooperative mean field decision problems. The readers are referred to (Caines, Huang, and Malhamé, 2017) for an overview. Most existing analysis has been developed based on two routes called the direct approach and the fixed point approach. By the direct approach, one starts by formally solving an $N$-player game to obtain a large coupled solution equation system, and next derives a simple limiting equation system by taking $N\to\infty$; see (Lasry and Lions, 2007) for the limit consisting of a Hamilton-Jacobi-Bellman (HJB) equation and a Fokker-Planck-Kolmogorov (FPK) equation. By the fixed point approach, one determines the best response of a representative agent to a mean field of an infinite population, and next all the agents’ best responses should regenerate that mean field (Huang, Malhamé, and Caines, 2006). This procedure formalizes a fixed point problem, which can be solved and further used to design decentralized strategies. For LQ mean field games, the recent work (Huang and Zhou, 2018b) shows the exact relationship of the two approaches. In general, the fixed point approach has more flexibilities and can be implemented in diverse models (Huang, Caines, and Malhamé, 2007; Li and Zhang, 2008; Bensoussan et al, 2013; Huang and Ma, 2016; Carmona and Delarue, 2018). Further convergence analysis in the direct approach can be found in (Cardaliaguet et al, 2015; Lacker, 2016; Fischer, 2017). Mean field games have found applications in traffic routing (Bauso, Zhang, and Papachristodoulou, 2017), smart grids (Couillet, et al, 2012; Ma, Callaway, and Hiskens, 2013; Kizilkale, Salhab, and Malhamé, 2019) and production planning (Wang and Huang, 2019), among others. A notable feature of the early literature of mean field games is that all players in the model are comparably small, and can be called peers.

Huang (2010) introduces an LQ mean field game model with a major player which has strong influence. A motivating example is the interaction between a large corporation and many much smaller firms. There has been a rapid increase of literature on mean field games with a major and many minor players. In the setting of LQ models, Nguyen and Huang (2012a) consider continuum parametrized minor players, and Nguyen and Huang (2012b) extend to mass behavior directly impacted by the major player. Kordonis and Papavassilopoulos (2015) analyze minor players with random entrance. Major players with leadership are studied by Bensoussan et al (2017), Moon and Basar (2018). Partial state observation is considered by Caines and Kizilkale (2017), Firoozi and Caines (2015). Huang, Wang and Wu (2016) take linear backward stochastic differential equations to model the dynamics of the players. Huang, Jaimungal, and Nourian (2015) present an application of the major player mean field game theory to an optimal execution model with an institutional trader and a large number of small traders.

Major-minor player games with nonlinear diffusion dynamics are an important class of modelling; see Nourian and Caines (2013), Buckdahn, Li and Peng (2014), Bensoussan, Chau and Yam (2016), Carmona and Zhu (2016). Leader-follower interaction is adopted by Bensoussan et al (2015), Fu and Horst (2018). To deal with this nonlinear modelling, forward-backward stochastic differential equations provide a vital analytical tool. Sen and Caines (2016) apply nonlinear filtering when the major player’s state is partially observed. More recently, Lasry and Lions (2018) introduce master equations for mean field games with major and minor players. They may be viewed as a pair of abstract dynamic programming equations. Cardadiaguet, Cirant, and Porretta (2018) prove the convergence of the Nash equilibria by use of the master equations when the number of minor players tends to infinity. A mean field principal-agent model is formulated by Elie, Mastrolia, and Possamai (2019). For major player models with discrete states, see (Huang 2012; Carmona and Wang, 2017; Kolokoltsov, 2017).

Huang (2010) applies a state space augmentation approach by adding the mean field dynamics into the two decision problems, one for the major player and one for a representative minor player. This Markovianizes the problem and enables the use of dynamic programming. The procedure of Huang (2010) is based on the fixed point approach and the associated consistent mean field approximations, and that work only assumes existence of the solution.

This paper analyzes the LQ mean field game with a major player and homogeneous (or symmetric) minor players and takes the direct approach by starting with the solution for $N+1$ players. Specifically, we will extend an asymptotic solvability notion introduced in a recent work Huang and Zhou (2018a) for LQ mean field games without a major player. With or without a major player, asymptotic solvability can be informally stated as the existence of Nash equilibria with complete state information for all sufficiently large population sizes, in addition to some boundedness property of the solution. We exploit the multi-scale nature of the optimization problem and use a re-scaling method in Huang and Zhou (2018a, 2018b) so that the key information in some higher order terms, as components in the solution matrices of $N+1$ coupled Riccati equations, can be captured. We derive the necessary and sufficient condition for asymptotic solvability and evaluate the value function. The re-scaling method gives a set of ordinary differential equations (ODEs) for nine matrix functions. To reveal the special structure underlying these functions, we will further relate them to the best responses of the major player and a representative minor player staying an infinite population, where consistent mean field approximations hold. The latter is a key feature of the fixed point approach in mean field games. Our mean field limit analysis shares similarity to (Cardadiaguet, Cirant, and Porretta, 2018) which performs convergence analysis in a nonlinear system via the master equation. But we explicitly exploit the multi-scale phenomena in our model to identify a lower dimensional object which governs the asymptotic behavior of the system when the number of minor players tends to infinity. Similar methods appear in the statistical physics literature on mean field models (Ott and Antonsen, 2008; Pazo and Montbrio, 2014).

We mention other related LQ models of finding mean field limits via analyzing large scale equations. Papavassilopoulos (2014) uses large algebraic Riccati equations in mean field games and analyzes existence by an implicit function theorem. Priuli (2015) considers coupled HJB and FPK equations with decentralized information. Mean field social optimal control is analyzed in (Huang 2003, Chap. 6; Herty, Pareschi, and Steffensen, 2015) via large Riccati equations.

The organization of the paper is as follows. Section 2 describes the LQ Nash game with $N+1$ players together with its solution via dynamic programming and Riccati equations. Section 3 extends the formulation of asymptotic solvability in Huang and Zhou (2018a, 2018b) to the LQ model with a major player. Section 4 presents further mean field limits and the performance. Section 5 formulates two optimal control problems under a mean field generated by an infinite number of minor players and addresses the relation to the asymptotic solvability problem. Numerical examples are presented in Section 6. Section 7 concludes the paper.

Notation: For symmetric matrix $S\geq 0$, we may write $x^{T}Sx=|x|_{S}^{2}$. For a matrix $Z=(z_{jk})\in\mathbb{R}^{l\times m}$, denote the $l_{1}$-norm $\|Z\|_{l_{1}}=\sum_{j,k}|z_{jk}|$. Let the function $g(\delta,x)$ be defined for $x$ in a subset $D_{g}$ of a Euclidean space and parameter $\delta\in(0,p]$ for some $p>0$. We say $g$ is compactly of $O(\delta)$ if for each compact subset $D_{0}\subset D_{g}$, there exists a constant $c_{0}$ depending on $D_{0}$ such that $\sup_{x\in D_{0}}|g(\delta,x)|\leq c_{0}\delta$.

2 The LQ game with major and minor players

We consider the LQ game with a major player ${\mathcal{A}}_{0}$ and $N$ minor players ${\mathcal{A}}_{i}$, $1\leq i\leq N$. At time $t\geq 0$, the states of ${\mathcal{A}}_{0}$ and ${\mathcal{A}}_{i}$ are, respectively, denoted by $X_{0}(t)$ and $X_{i}(t)$, $1\leq i\leq N$. The dynamics of the $N+1$ players are given by a system of linear stochastic differential equations (SDEs):

[TABLE]

where we have state $X_{i}\in\mathbb{R}^{n}$, control $u_{i}\in\mathbb{R}^{n_{1}}$, and $X^{(N)}=\frac{1}{N}\sum_{k=1}^{N}X_{k}(t)$. The initial states $\{X_{j}(0),0\leq j\leq N\}$ are independent with $EX_{j}(0)=x_{j}(0)$ and finite second moment. The $N+1$ standard $n_{2}$-dimensional Brownian motions $\{W_{j},0\leq j\leq N\}$ are independent and also independent of the initial states. The deterministic constant matrices $A_{0}$, $A$, $B_{0}$, $B$, $D_{0}$, $D$, $F_{0}$, $F$, $G$ have compatible dimensions. Denote $u=(u_{0},\cdots,u_{N})$. The costs of players ${\mathcal{A}}_{k}$, $0\leq k\leq N$, are given by

[TABLE]

The deterministic constant matrices (or vectors) $Q_{0}$, $\Gamma_{0}$, $\eta_{0}$, $R_{0}$, $Q_{0f}$, $\Gamma_{0f}$, $\eta_{0f}$, $Q$, $\Gamma_{1}$, $\Gamma_{2}$, $\eta$, $R$, $Q_{f}$, $\Gamma_{1f}$, $\Gamma_{2f}$, $\eta_{f}$ above have compatible dimensions, and $Q_{0}\geq 0$, $Q_{0f}\geq 0$, $Q\geq 0$, $Q_{f}\geq 0$, $R_{0}>0$, $R>0$. For notational simplicity, we only consider constant parameters. Our analysis can be easily extended to the case of time-dependent parameters. Define

[TABLE]

We denote by ${\bf 1}_{k\times l}$ a $k\times l$ matrix with all entries equal to 1, and by the column vectors $\{e_{1}^{k},\ldots,e_{k}^{k}\}$ the canonical basis of $\mathbb{R}^{k}$. For instance, $e_{1}^{k}=[1,0,\cdots,0]^{T}\in\mathbb{R}^{k}$. For matrices $K=(k_{ij})\in\mathbb{R}^{l_{1}\times l_{2}}$, $\hat{K}\in\mathbb{R}^{l_{3}\times l_{4}}$, the Kronecker product $K\otimes\hat{K}=(k_{ij}\hat{K})_{1\leq i\leq l_{1},1\leq j\leq l_{2}}\in\mathbb{R}^{(l_{1}l_{3})\times(l_{2}l_{4})}$. We may use a subscript $n$ to indicate the identity matrix $I_{n}$ to be $n\times n$.

Now we write (1) and (2) in the form

[TABLE]

where $s\geq 0$. We consider closed-loop perfect state (CLPS) information so that $X(s)$ is observed by each player, and look for Nash strategies in this section. Let $u_{-k}$ denote the strategies of all players other than ${\mathcal{A}}_{k}$. A set of strategies $(\hat{u}_{0},\cdots,\hat{u}_{N})$ is a Nash equilibrium if for any $0\leq k\leq N$, we have

[TABLE]

for any state feedback based strategy $u_{k}$ which together with $\hat{u}_{-k}$ ensures a unique solution of $X(s)$ on $[0,T]$. Denote the value function of ${\mathcal{A}}_{j}$ by $V_{j}(t,\mathbold{x})$, $0\leq j\leq N$, which corresponds to the initial time-state pair $(t,\mathbold{x})$ in (5), i.e., $X(t)=\mathbold{x}$ at the initial time $t$, and can be interpreted as $J_{j}$ evaluated on the time interval $[t,T]$ under the set of Nash strategies. The set of value functions is determined by the system of HJB equations

[TABLE]

and

[TABLE]

where $\mathbold{x}^{(N)}=(1/N)\sum_{i=1}^{N}\mathbold{x}_{i}$ and the minimizers are

[TABLE]

Next we substitute $u_{0}$ and $u_{i}$ into (2) and (2):

[TABLE]

and

[TABLE]

Suppose $V_{j}(t,\mathbold{x}),~{}0\leq j\leq N,$ has the following form

[TABLE]

where $\mathbold{P}_{j}(t)$ is symmetric. Then

[TABLE]

Denote

[TABLE]

where $I_{n}$ is the $(i+1)$th submatrix in (14). We have $\mathbold{K}_{0},\mathbold{K}_{i}\in\mathbb{R}^{n\times(N+1)n}$ and $\mathbold{Q}_{0},\mathbold{Q}_{i}\in\mathbb{R}^{(N+1)n\times(N+1)n}$. We write

[TABLE]

We may write $V_{j}(T,\mathbold{x})$, $0\leq j\leq N$, in a similar form.

We substitute (13) into (2) and derive the equation systems:

[TABLE]

[TABLE]

[TABLE]

By (2) and (13), we derive the equation systems:

[TABLE]

[TABLE]

[TABLE]

Remark 1**.**

If (17) and (20) have a solution $(\mathbold{P}_{0},\cdots,\mathbold{P}_{N})$ on $[\tau,T]\subseteq[0,T]$, such a solution is unique due to the local Lipschitz continuity of the vector field; see (Hale, 1969). The ODE guarantees each $\mathbold{P}_{j}$, $0\leq j\leq N$, to be symmetric. If (17) and (20) have a unique solution $(\mathbold{P}_{0},\cdots,\mathbold{P}_{N})$ on $[0,T]$, then we can uniquely solve $(\mathbold{S}_{0},\cdots,\mathbold{S}_{N})$ and $(\mathbold{r}_{0},\cdots,\mathbold{r}_{N})$.

Lemma 1**.**

Suppose that (17) and (20) have a unique solution $(\mathbold{P}_{0},\cdots,\mathbold{P}_{N})$ on $[0,T]$. Then we can uniquely solve (18), (19), (21), (22), and the Nash game of $N+1$ players has a set of feedback Nash strategies given by

[TABLE]

Proof. This lemma follows the standard results in (Basar and Olsder, 1999, Theorem 6.16, Corollary 6.5). $\Box$

By Lemma 1 and Remark 1, the solution of the feedback Nash strategies completely reduces to the study of (17) and (20).

3 Asymptotic solvability

Define the $(N+1)n\times(N+1)n$ identity matrix

[TABLE]

For $1\leq i\neq j\leq N+1$, exchanging the $i$th and $j$th rows of submatrices in $I_{(N+1)n}$, let $J_{ij}$ denote the resulting matrix. For instance, we have

[TABLE]

It is easy to check that $J_{ij}^{T}=J_{ij}^{-1}=J_{ij}$.

Theorem 2**.**

$\mathbold{P}_{0}(t)$* and $\mathbold{P}_{1}(t)$ have the representation:*

[TABLE]

and

[TABLE]

where each submatrix depends on $t$ and is $n\times n$. Moreover, ${\mathbold{P}}_{i}(t)=J_{2,i+1}^{T}{\mathbold{P}}_{1}(t)J_{2,i+1}$ for $i\geq 2$.

Proof. See Appendix A. $\Box$

Definition 3**.**

The sequence of Nash games (1)-(4) has asymptotic solvability if there exists $N_{0}$ such that for all $N\geq N_{0}$, $(\mathbold{P}_{0},\cdots,\mathbold{P}_{N})$ has a solution on $[0,T]$ and

[TABLE]

$\Box$**

Note that (27) is equivalent to

[TABLE]

Denote

[TABLE]

Define

[TABLE]

For (17) and (20) we write the ODE system for the set of variables $(\Lambda_{1}^{0N},\Lambda_{2}^{0N},\dots,\Lambda_{b}^{N})$ in (33); see Appendix B. The following ODE system is obtained as the limit of the above ODE system with respect to $N$:

[TABLE]

where the terminal conditions are

[TABLE]

Theorem 4**.**

The sequence of games in (1)-(4) has asymptotic solvability if and only if (56) has a solution on $[0,T]$.

Proof. See Appendix B. $\Box$

Due to the quadratic terms in its right hand sides, we call (56) a system of Riccati ODEs. As it turns out later in Section 5, this set of solution functions can be interpreted according to two optimal control problems.

4 Equilibrium costs and decentralized control

For this section, we assume (56) has a solution on $[0,T]$. Therefore there exists $N_{0}>0$ such that for all $N\geq N_{0}$, (17) and (20) have a solution $(\mathbold{P}_{0},\cdots,\mathbold{P}_{N})$ on $[0,T]$.

Proposition 5**.**

Let $(\mathbold{S}_{0},\cdots,\mathbold{S}_{N})$ be the solution of (18) and (21). We have the representation

[TABLE]

where each vector of ${\theta_{0}^{0}(t),~{}\theta_{1}^{0}(t),~{}\theta_{0}(t),~{}\theta_{1}(t),~{}\theta_{2}(t)}$ is in $\mathbb{R}^{n}$ and $\theta_{1}^{T}$ is the $(i+1)$th component of $\mathbold{S}_{i}$.

Proof. The method is similar to proving Theorem 2, and we omit the detail. $\Box$

Define

[TABLE]

We derive a set of ODEs for $(\alpha_{0}^{0N},\alpha_{1}^{0N},\alpha_{0}^{N},\alpha_{1}^{N},\alpha_{2}^{N})$; see Appendix C. By taking the limit form of these equations with respect to $N$, we introduce the ODE system:

[TABLE]

where

[TABLE]

After (56) is solved, $(\alpha_{0}^{0},\cdots,\alpha_{2})$ satisfies a linear ODE system and can be uniquely solved on $[0,T]$.

Proposition 6**.**

We have

[TABLE]

Proof. We consider the first ODE system for $(\Lambda_{1}^{0N},\cdots,\Lambda_{b}^{N})$ and $(\alpha_{0}^{0N},\cdots,\alpha_{2}^{N})$, and the second ODE system for $(\Lambda_{1}^{0},\ldots,\Lambda_{b})$ and $(\alpha_{0}^{0},\cdots,\alpha_{2})$. By (Huang and Zhou, 2018b, Theorem 4), we obtain the error bound. $\Box$

In view of (19) and (22), we obtain

[TABLE]

where ${\mathbold{r}}_{0}(T)=\eta_{0f}^{T}Q_{0f}\eta_{0f}$ and ${\mathbold{r}}_{i}(T)=\eta_{f}^{T}Q_{f}\eta_{f}$. For $N\geq N_{0}$, we can uniquely solve ${\mathbold{r}}_{0}$ and ${\mathbold{r}}_{i}$. It is clear that ${\mathbold{r}}_{i}$ does not depend on $i$. We rewrite

[TABLE]

As the approximation of (63) and (64), we introduce the ODE system

[TABLE]

where $\chi_{0}(T)=\eta_{0f}^{T}Q_{0f}\eta_{0f}$ and $\chi(T)=\eta_{f}^{T}Q_{f}\eta_{f}$, and we solve $(\chi_{0},~{}\chi)$ on $[0,~{}T]$.

Proposition 7**.**

We have

[TABLE]

Proof. The proof is similar to that of Proposition 6. $\Box$

Assumption (H): The initial states $X_{1}(0),X_{2}(0),\cdots$, are i.i.d. and $X_{1}(0)$ has mean $\mu$, and covariance $\Sigma$. In addition, $X_{0}(0)$ has mean $\mu_{0}$ and covariance $\Sigma_{0}.$

Denote the set of Nash strategies $\hat{u}=(\hat{u}_{0},\hat{u}_{1},\cdots,\hat{u}_{N})$ given by (23)-(24).

Proposition 8**.**

Under Assumption (H), the costs under the set of strategies $\hat{u}$ have the asymptotic form

[TABLE]

[TABLE]

Proof. Note that

[TABLE]

and

[TABLE]

where $X_{-0}(0)=[X_{1}^{T}(0),X_{2}^{T}(0),\cdots,X_{N}^{T}(0)]^{T}$. Similarly we have

[TABLE]

We complete the proof by elementary computations and taking limits. $\Box$

Substituting $\hat{u}_{0}$ and $\hat{u}_{i}$ into (1) and (2), we have

[TABLE]

When $N\to\infty$, we obtain a limit form of the strategies

[TABLE]

where $\overline{X}$ is the infinite population limit of the state average ${X}^{(N)}$ of the minor players in (69). For the $N+1$ player game, we replace $\overline{X}$ in the strategies (70)-(71) by $\overline{X}^{\dagger}$ and write the closed-loop system of equations:

[TABLE]

where $\overline{X}^{\dagger}$ is generated by the $N+1$ players instead of an infinite population. Denote the strategies in (72)-(73) by $(\check{u}_{0}^{\dagger},\cdots,\check{u}_{N}^{\dagger})$. Following the standard mean square error estimate of $|X^{(N)}-\overline{X}|$ for (72)-(73), under assumption (H) we can show that $(\check{u}_{0}^{\dagger},\cdots,\check{u}_{N}^{\dagger})$ is an $\epsilon$-Nash equilibrium for the $N+1$ player game, where $\epsilon=O(1/\sqrt{N})$ and each player may use centralized state information $X(t)$; see related methods in (Huang, 2010).

By the re-scaling technique, we derive the mean field limits of the costs and strategies. The feasibility condition is determined by (56) directly based on the model parameters in (1)-(4). This is different from (Huang, 2010), where the existence condition is described in an augmented state space in $3n$ dimensions and imposes consistency requirements on $3n\times 3n$ matrices.

5 The limiting control problems and best responses

For this section, we assume (56) has a solution on $[0,T]$.

An interesting question is whether the above two limit strategies in (70)-(71) have the interpretation as best responses in appropriately constructed optimal control problems. Finding the best response of a single agent in an infinite population model has been a key step in the fixed point approach in mean field games; see (Huang, Caines, and Malhamé, 2007; Huang, 2010). We introduce two optimal control problems.

Problem (P0): The dynamics are given by

[TABLE]

where $X_{0}(0)$ and $\overline{X}(0)=\mu_{0}$ are given. Equation (75) may be viewed as the limit of (69) but now $X_{0}$ is indirectly controlled by $u_{0}$ in (74). The cost is

[TABLE]

Problem (P1): The dynamics are given by

[TABLE]

where $X_{1}(0),~{}X_{0}(0)$ and $\overline{X}(0)=\mu_{0}$ are given. The notation $(X_{0},\overline{X})$ is reused in Problem (P1), where $u_{0}$ has taken a specific form in (76). Equation (76) can be viewed as a limit form of (72) when $N\to\infty$. Since the two problems will be solved separately, this should cause no risk of confusion. The cost is

[TABLE]

Since $R_{0}>0,~{}Q_{0},~{}Q_{0,f}\geq 0,~{}R>0,~{}Q,~{}Q_{f}\geq 0$, both Problem (P0) and Problem (P1) can be solved. The resulting optimal control laws will also be called best responses.

Below, we start with the solution of Problem (P0). Denote

[TABLE]

Then we have

[TABLE]

We further write

[TABLE]

By dynamic programming for the optimal control problem (P0), we introduce

[TABLE]

where the terminal condition can be determined as

[TABLE]

We uniquely solve ${\mathbb{P}}_{0}$ and ${\mathbb{S}}_{0}$ on $[0,T]$. Note that $\mathbb{P}_{0}$ is a $2n\times 2n$ matrix. The optimal control law is

[TABLE]

Denote

[TABLE]

where $\Phi_{1}^{0}$ and $\Phi_{3}^{0}$ are symmetric. Then by (97), we derive the ODE system:

[TABLE]

where

[TABLE]

And by (102),

[TABLE]

where

[TABLE]

Finally, we rewrite $u_{0}^{*}$ as

[TABLE]

Now we give the solution of Problem (P1). Denote

[TABLE]

The dynamics can be given as

[TABLE]

By dynamic programming, we introduce the two ODEs:

[TABLE]

where

[TABLE]

We uniquely solve ${\mathbb{P}}$ and ${\mathbb{S}}$ on $[0,T]$. Denote

[TABLE]

where $\Phi_{0}$, $\Phi_{1}$ and $\Phi_{3}$ are symmetric. By (122), we derive the ODE system:

[TABLE]

where

[TABLE]

Now by (129), we derive

[TABLE]

where

[TABLE]

The optimal control law is given by

[TABLE]

Theorem 9**.**

We have

[TABLE]

and

[TABLE]

Proof. We can directly show that

[TABLE]

is a solution of (56). Similarly, $(\beta_{0}^{0},\beta_{1}^{0},\beta_{0},\beta_{1},\beta_{2})$ satisfies the ODE of $(\alpha_{0}^{0},\alpha_{1}^{0},\alpha_{0},\alpha_{1},\alpha_{2})$. Therefore, we obtain the representation of $(\mathbb{P}_{0},\mathbb{P},\mathbb{S}_{0},\mathbb{S})$. $\Box$

By Theorem 9, after appropriate arrangement the matrix functions in the solution of (56) have the interpretation as the solutions of two Riccati-like equations.

It is now clear that $(u_{0}^{*},u_{1}^{*})$ agrees with $(\check{u}_{0},\check{u}_{i})$ given by (70)-(71). Then we have the following interpretation on $(\check{u}_{0},\check{u}_{i})$ within an infinite population of minor players. First, $\check{u}_{0}$ is the best response with respect to $\overline{X}$; second, $\check{u}_{i}$ is the best response with respect to $({\overline{X}},x_{0},\check{u}_{0})$; and finally, $\overline{X}$ is generated by the infinite number of minor players applying their best responses. This suggests a consistent mean field approximation, which is well known in the fixed point approach of mean field games (Caines, Huang, and Malhamé, 2017). In our mean field limit here, the consistent mean field approximation is a derived property. This is in contrast to the major player model in (Huang, 2010; Carmona and Zhu, 2016), where consistent mean field approximations are imposed as a requirement at the beginning so that individual strategies can be determined.

6 Numerical examples

We have seen that testing asymptotic solvability reduces to checking the solution of (56) on $[0,T]$. It is generally infeasible to solve (56) analytically. Its numerical solution provides a practical means to check asymptotic solvability.

Example 1**.**

The parameters in (1)-(4) are given by $A_{0}=1$, $B_{0}=2$, $F_{0}=0.5$, $A=0.5$, $B=1$, $F=0.2$, $G=0.4$, $Q_{0}=1$, $R_{0}=0.5$, $Q=2$, $R=1$, $\Gamma_{0}=0.8$, $\Gamma_{1}=0.3$, $\Gamma_{2}=0.5$, $Q_{0f}=Q_{f}=0$, and $T=12$. We use ode45 of MatLab to numerically solve (56) on $[0,T]$; see Fig. 1. The existence of the solution suggests asymptotic solvability holds.

Example 2**.**

$A_{0}=0.3$, $B_{0}=1$, $F_{0}=0.2$, $A=0.2$, $B=1$, $F=1$, $G=-0.2$, $Q_{0}=2$, $R_{0}=1$, $Q=1$, $R=1$, $\Gamma_{0}=0.8$, $\Gamma_{1}=0.1$, $\Gamma_{2}=1.2$, $Q_{0f}=Q_{f}=0$, and $T=2.5$. The numerical solution of (56) has a finite escape time between $0.5$ and $1$ as shown in Fig. 2, suggesting no asymptotic solvability.

7 Concluding remarks

We study an asymptotic solvability problem for LQ Nash games involving a major player and $N$ minor players, where $N$ tends to infinity. We obtain the necessary and sufficient condition of asymptotic solvability via a system of Riccati ODEs and evaluate the equilibrium costs. The system of Riccati ODEs has close relation with a limiting control model of two players: the major player and a representative minor player. For future work, it is of interest to generalize our analysis to deal with leadership (Bensoussan et al, 2017), noisy measurements (Firoozi and Caines, 2015), and control constraints (Hu, Huang and Li, 2018).

Appendix A: Proof of Theorem 2

Lemma A.1**.**

Assume that (17) and (20) have a solution $(\mathbold{P}_{0}(t),\cdots,\mathbold{P}_{N}(t))$ on $[0,T]$. Then the following holds.

i) ${\mathbold{P}}_{0}(t)$ has the representation

[TABLE]

where $\Pi_{1}^{0}(t)$, $\Pi_{3}^{0}(t)$ and $\Pi_{4}^{0}(t)$ are $n\times n$ symmetric matrix functions. The matrix $\Pi_{4}^{0}$ appears for $N^{2}-N$ times.

ii) ${\mathbold{P}}_{1}(t)$ has the representation

[TABLE]

where $\Pi_{0}(t)$, $\Pi_{1}(t)$, $\Pi_{3}(t)$ and $\Pi_{4}(t)$ are $n\times n$ symmetric matrix functions. The matrix $\Pi_{4}$ appears for $(N-1)(N-2)$ times.

iii) For $i>1$, $\mathbold{P}_{i}(t)=J_{2,i+1}^{T}\mathbold{P}_{1}(t)J_{2,i+1}$.

Proof. i) For $0\leq l\leq N$, denote $\mathbold{P}_{l}=(\mathbold{P}_{l}^{jk})_{1\leq j,k\leq N+1},$ where $\mathbold{P}_{l}^{jk}$ is an $n\times n$ matrix. Let $\hat{\mathbold{P}_{l}}=J_{23}^{T}\mathbold{P}_{l}J_{23}$. By the method in (Huang and Zhou, 2018b, Lemma A.1) and elementary matrix computations, we can verify that $(\hat{\mathbold{P}_{0}},\hat{\mathbold{P}_{2}},\hat{\mathbold{P}_{1}},\hat{\mathbold{P}_{3}},\cdots,\hat{\mathbold{P}_{N}})$ satisfies (17) and (20). Hence,

[TABLE]

Then ${\mathbold{P}_{2}}=J_{23}^{T}\mathbold{P}_{1}J_{23}$ and ${\mathbold{P}_{0}}=J_{23}^{T}\mathbold{P}_{0}J_{23}$, and we obtain $\mathbold{P}_{0}^{22}=\mathbold{P}_{0}^{33},~{}\mathbold{P}_{0}^{12}=\mathbold{P}_{0}^{13}$.

Taking $J_{k,k+1},~{}k\geq 3$ in place of $J_{23}$ and following the method in (Huang and Zhou, 2018b), we obtain the representation of $\mathbold{P}_{0}$.

ii) Now denote $\hat{\mathbold{P}_{l}}=J_{34}^{T}\mathbold{P}_{l}J_{34}$, and we can verify that

[TABLE]

is a solution of (17) and (20). Hence,

[TABLE]

This yields $\mathbold{P}_{1}^{13}=\mathbold{P}_{1}^{14}$, $\mathbold{P}_{1}^{23}=\mathbold{P}_{1}^{24}$ and $\mathbold{P}_{1}^{33}=\mathbold{P}_{1}^{44}$. In addition, $\mathbold{P}_{1}^{43}=\mathbold{P}_{1}^{34}$. Since $\mathbold{P}_{1}$ is symmetric, $(\mathbold{P}_{1}^{43})^{T}=\mathbold{P}_{1}^{34}$. So $\mathbold{P}_{1}^{34}$ is symmetric. Similarly, by the relation $J_{45}^{T}\mathbold{P}_{1}J_{45}=\mathbold{P}_{1},$ we obtain $\mathbold{P}_{1}^{34}=\mathbold{P}_{1}^{35}$. Now, repeatedly using the relation $J_{k,k+1}^{T}\mathbold{P}_{1}J_{k,k+1}=\mathbold{P}_{1}$ for all $k\geq 3$, we obtain the representation of $\mathbold{P}_{1}$. Note that $\Pi_{4}$ is symmetric.

iii) This equality can be shown as in the case $i=2$ in the proof of part i). $\Box$

Proof of Theorem 2: By Lemma A.1, we have

[TABLE]

and

[TABLE]

and

[TABLE]

and

[TABLE]

We have $\Pi_{3}^{0}(T)=\Pi_{4}^{0}(T)$, and

[TABLE]

It follows that

[TABLE]

By Lemma A.1, we have

[TABLE]

and

[TABLE]

and

[TABLE]

and

[TABLE]

and

[TABLE]

and

[TABLE]

and

[TABLE]

We have $\Pi_{3}(T)=\Pi_{4}(T)$, and

[TABLE]

Therefore, $\Pi_{3}(t)=\Pi_{4}(t)$ for all $t\in[0,T]$. This completes the proof. $\Box$

Appendix B: Proof of Theorem 4

Step 1. By (A.1), (Appendix A: Proof of Theorem 2) and (A.5), we determine

[TABLE]

[TABLE]

[TABLE]

where $\Lambda_{1}^{0N}(T)=Q_{0f}$, $\Lambda_{2}^{0N}(T)=-Q_{0f}\Gamma_{0f}$, $\Lambda_{3}^{0N}(T)=\Gamma_{0f}^{T}Q_{0f}\Gamma_{0f}$, and

[TABLE]

For reasons of space, the expression of $g_{3}^{0}$ is not displayed. We further obtain

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

where $g_{0},\cdots,g_{b}$ are not displayed and are compactly of $O(1/N)$.

Step 2. Proof of Theorem 4.

Denote $\xi^{N}=(\Lambda_{1}^{0N},\Lambda_{2}^{0N},\cdots,\Lambda_{b}^{N})$ for (33), and we view each of $g_{2}^{0}$, $g_{3}^{0}$, $g_{0}$, $g_{1}$, $g_{2}$, $g_{3}$, $g_{a}$, $g_{b}$ as a function of $\xi^{N}$ with parameter $1/N$. They are all compactly of $O(1/N)$. For some $C>0$, we further have

[TABLE]

Subsequently, we view the ODE system (B.1)-(B.9) as a slightly perturbed form of (56). The remaining proof is similar to that of (Huang and Zhou, 2018b, Theorem 5) and we only give its sketch. If asymptotic solvability holds, we solve (B.1)-(B.9) for all sufficiently large $N$. By taking some increasing subsequence of population sizes $N_{1}<N_{2}<\cdots$, we can ensure that as $k\to\infty$, their solutions $\{\xi^{N_{k}}(t),k=1,2,\cdots\}$ have a limit as a vector function on $[0,T]$ which satisfies the limit ODE (56) on $[0,T]$. Conversely, if (56) has a solution on $[0,T]$, there exists $N_{0}>0$ such that (B.1)-(B.9) has a solution on $[0,T]$ for all $N\geq N_{0}$; all these solutions are uniformly bounded. Accordingly we obtain $(\mathbold{P}_{0},\ldots,\mathbold{P}_{N})$ to satisfy (27) for all $N\geq N_{0}$. So asymptotic solvability holds.  $\Box$

Appendix C

In view of (18) and (21), by Proposition 5 we have

[TABLE]

where

[TABLE]

and

[TABLE]

where

[TABLE]

By (C.5), we have the relation

[TABLE]

where $\alpha_{0}^{0N}(T)=-Q_{0f}\eta_{0f}$, $\alpha_{1}^{0N}(T)=\Gamma_{0f}^{T}Q_{0f}\eta_{0f},$ and $h_{1}^{0}(1/N,\alpha_{1}^{0N},\Lambda_{2}^{N})=-\tfrac{1}{N}{\Lambda_{2}^{N}}^{T}\alpha_{1}^{0N}.$ By (C.15), we have

[TABLE]

where the terminal condition is

[TABLE]

and $h_{0}$, $h_{1}$, $h_{2}$ are compactly of $O(1/N)$.