Stationary Equilibria of Mean Field Games with Finite State and Action Space

TL;DR

This paper studies stationary equilibria in finite state and action mean field games with Markovian dynamics, providing existence results, characterization methods, and computational techniques for these equilibria.

Contribution

It introduces a framework for finite state and action mean field games with Markov dynamics, establishing existence and characterization of stationary equilibria.

Findings

Existence of stationary mean field equilibria under mild conditions.

Characterization of equilibria as fixed points of a transition-based map.

Demonstration of techniques on two example models.

Abstract

Mean field games formalize dynamic games with a continuum of players and explicit interaction where the players can have heterogeneous states. As they additionally yield approximate equilibria of corresponding $N$-player games, they are of great interest for socio-economic applications. However, most techniques used for mean field games rely on assumptions that imply that for each population distribution there is a unique optimizer of the Hamiltonian. For finite action spaces, this will only hold for trivial models. Thus, the techniques used so far are not applicable. We propose a model with finite state and action space, where the dynamics are given by a time-inhomogeneous Markov chain that might depend on the current population distribution. We show existence of stationary mean field equilibria in mixed strategies under mild assumptions and propose techniques to compute all these equilibria. More precisely, our results allow -- given that the generators are irreducible -- to characterize the set of stationary mean field equilibria as the set of all fixed points of a map completely characterized by the transition rates and rewards for deterministic strategies. Additionally, we propose several partial results for the case of non-irreducible generators and we demonstrate the presented techniques on two examples.

Equations

$${\left(Q^{\pi }(m(t),t)\right)}_{ij}=\sum _{a\in \mathcal{A}}{Q}_{ija}(m(t)){\pi }_{ia}(t),$$

$$V{}_{x_0}({\pi }^0,m)={\int }_0^{\infty }\left(\sum _{i\in \mathcal{S}}\sum _{a\in \mathcal{A}}{x}_i^{{\pi }_0}(t)r_{ia}(m(t)){\pi }_{i,a}^0(t)\right)e^{-\beta t}\text{d}t,$$

$${\dot{m}}_j(t)=\sum _{i\in \mathcal{S}}{m}_i(t)Q_{ij}^{\pi }(m(t),t)\forall j\in \mathcal{S}$$

$$0=\sum _{i\in \mathcal{S}}{m}_i{Q}_{ij}^{\pi }(m)\forall j\in \mathcal{S}.$$

$$V{}_{x_0}({\pi }^0,m)$$

$$={\int }_0^{\infty }\sum _{i\in \mathcal{S}}\sum _{a\in \mathcal{A}}\sum _{k\in \mathcal{S}}{x}_k^0{p}^{{\pi }^0}(0,k,t,i)r_{i,a}(m){\pi }_{i,a}^0(t)e^{-\beta t}\text{d}t$$

$$=\sum _{k\in \mathcal{S}}{x}_k^0{\int }_0^{\infty }{e}^{-\beta t}\sum _{i\in \mathcal{S}}\sum _{a\in \mathcal{A}}{r}_{i,a}(m){\pi }_{ia}^0(t)p^{{\pi }^0}(0,k,t,i)\text{d}t$$

$$=\sum _{k\in \mathcal{S}}{x}_k^0{\int }_0^{\infty }{e}^{-\beta t}{\mathbb{E}}_k^{{\pi }^0}\left[r(x(t),{\pi }_t^0)\right]\text{d}t$$

$$=\sum _{k\in \mathcal{S}}{x}_k^0{V}_k^{{\pi }^0}(m),$$

$$\mathcal{D}(m)=\{d:\mathcal{S}\to \mathcal{A}:d(i)\in O_i(m)\text{ for all }i\in \mathcal{S}\}$$

$$O_i(m)={\text{argmax}}_{a\in \mathcal{A}}\left\{r_{ia}(m)+\sum _{j\in \mathcal{S}}{Q}_{ija}(m)V_j^{\ast }(m)\right\},$$

$${\bar{P}}_{ija}(m)=\frac{Q_{ija}(m)}{||Q(m)||}+{\delta }_{ij}$$

$${\bar{r}}_{ia}(m)=\frac{r_{ia}(m)}{\beta +||Q(m)||}.$$

$$O_i(m)={\text{argmax}}_{a\in \mathcal{A}}\left\{{\bar{r}}_{ia}(m)+\alpha {\bar{P}}_{ija}(m){\tilde{V}}_j^{\ast }(m)\right\}$$

$$\pi =\sum _{i=1}^n{\lambda }_i{d}_i\text{with}{\lambda }_i\ge 0\forall i\in \{1,\dots ,n\}\text{and}\sum _{i=1}^n{\lambda }_i=1.$$

$${\bar{r}}^{\pi }(m{)}_i=\sum _{a\in \mathcal{A}}{\pi }_{ia}{\bar{r}}_{ia}(m)\text{ and }({\bar{P}}^{\pi }{)}_{ij}=\sum _{a\in \mathcal{A}}{\pi }_{ia}{\bar{P}}_{ija}(m).$$

$$V^{\pi }={\bar{r}}^{\pi }+\alpha {\bar{P}}^{\pi }{V}^{\pi }=\sum _{i=1}^n{\lambda }_i({\bar{r}}^{d_i}+\alpha {\bar{P}}^{d_i}{V}^{\pi }).$$

$$r^{\pi }+\alpha P^{\pi }{V}^{\ast }=\sum _{i=1}^n{\lambda }_i(r^{d_i}+\alpha P^{d_i}{V}^{\ast })=\sum _{i=1}^n{\lambda }_i{V}^{\ast }=V^{\ast }.$$

$$\pi =\sum _{d\in D^s}{\lambda }_{d}d\text{with}{\lambda }_d={\pi }_{1d(1)}\cdot \dots \cdot {\pi }_{Sd(S)}.$$

$$V^{\pi }(i)$$

$$=\sum _{a\in \mathcal{A}}{\pi }_{ia}\left({\bar{r}}_{ia}(m)+\sum _{j\in S}\alpha {\bar{P}}_{ija}(m)V^{\pi }(j)\right)$$

$$\le \sum _{a\in \mathcal{A}}{\pi }_{ia}\left({\bar{r}}_{ia}(m)+\sum _{j\in S}\alpha {\bar{P}}_{ija}(m)V^{\ast }(j)\right)$$

$$<\sum _{a\in \mathcal{A}}{\pi }_{ia}\underset{a^{\prime }\in A_i}{\max }\left\{{\bar{r}}_{i{a}^{\prime }}(m)+\sum _{j\in S}\alpha {\bar{P}}_{ij{a}^{\prime }}(m)V^{\ast }(j)\right\}$$

$$=\underset{a^{\prime }\in \mathcal{A}}{\max }\left\{{\bar{r}}_{i{a}^{\prime }}(m)+\sum _{j\in S}\alpha {\bar{P}}_{ij{a}^{\prime }}(m)V^{\ast }(j)\right\}=V^{\ast }(i).$$

$$V^d(m)=(\beta I-Q^d(m){)}^{-1}{r}^d(m)$$

$$\varphi (m):=\{x\in \mathcal{P}(\mathcal{S})|\exists \pi \in \text{conv}(\mathcal{D}(m)):0=x^T{Q}^{\pi }(m)\}.$$

$$0=\sum _{i\in \mathcal{S}}\sum _{a\in \mathcal{A}}{x}_i^1{Q}_{ija}(m){\pi }_{ia}^1\text{and}0=\sum _{i\in \mathcal{S}}\sum _{a\in \mathcal{A}}{x}_i^2{Q}_{ija}(m){\pi }_{ia}^2.$$

$$0=\sum _{i\in \mathcal{S}}\sum _{a\in \mathcal{A}}{Q}_{ija}(m)z_{ia}^1\text{and}0=\sum _{i\in \mathcal{S}}\sum _{a\in \mathcal{A}}{Q}_{ija}(m)z_{ia}^2.$$

$$0=\sum _{i\in \mathcal{S}}\sum _{a\in \mathcal{A}}{Q}_{ija}(m)z_{ia}^3,$$

$${\pi }_{ia}^3:=\{\begin{array}{ll}0& \text{if }{x}_i^3=0\\ z_{ia}^3/x_i^3& \text{if }{x}_i^3>0\end{array}$$

$$0=\sum _{i\in \mathcal{A}}\sum _{a\in \mathcal{A}}{x}_i^{n_k}{Q}_{ija}(m^{n_k}){\pi }_{ia}^{n_k}\leftarrow \sum _{i\in \mathcal{A}}\sum _{a\in \mathcal{A}}{x}_i{Q}_{ija}(m){\pi }_{ia},$$

$$\sum _{j\in \mathcal{T}}\sum _{i\in \mathcal{S}\setminus \mathcal{T}}{m}_i{Q}_{ij}^{\pi }(m)=\sum _{j\in \mathcal{T}}\sum _{i\in \mathcal{S}\setminus \mathcal{T}}{m}_j{Q}_{ji}^{\pi }(m).$$

$$0=\sum _{i\in \mathcal{S}}{m}_i{Q}_{ij}^{\pi }(m),$$

$$m_j\sum _{i\in \mathcal{S}}{Q}_{ji}^{\pi }(m)=0.$$

Full text

1 Introduction

Mean field games have been introduced independently by Lasry and Lions (2007) and Huang et al. (2006) in order to provide a framework for dynamic stochastic games in continuous time with a continuum of players, whose equilibria furthermore serve as approximate Nash equilibria for corresponding $N$-player games. The main feature of these games is that any player does not observe the state and action of each other player individually, but only at an aggregated level through the empirical distribution of these characteristics.

From an economic perspective these games are of particular interest as they yield a formal way to describe games with a continuum of rational players that accounts for explicit interaction (in contrast to the classical assumption in general equilibrium theory that “prices mediate all social interaction” (Guéant et al., 2011)) as well as heterogeneity of states (in contrast to representative agent models (see Gomes et al. (2015, p.209))). Therefore a wide range of economic models relying on mean field games emerged, which includes growth models, the production of an exhaustible resource by a continuum of producers as well as opinion dynamics (see Gomes et al. (2015), Guéant et al. (2011), Caines et al. (2017) and the references therein).

In classical mean field games models each individual player’s dynamics is given by a diffusion process whose drift and volatility depend on time, the current state and the current action of the individual player as well as the current distribution of all players. Each player individually solves an optimal control problem given these dynamics, where the costs also depend on the current state and action of the individual player as well as distribution of all players. A mean field equilibrium is then given by a flow of population distributions $m$ such that there is an optimal strategy $\pi$ for the individual control problem given $m$ and the distribution of the individual player given this strategy is in turn $m$. This is a natural analogue of a Nash equilibrium of games with $N$ players: Given that all players play the strategy $\pi$, no player wants to deviate from playing $\pi$, as the population distribution is $m$ and $\pi$ is a best response to it.

To solve this type of mean field games one then sets up several assumptions, which usually include the assumption that there is a unique optimizer of the Hamiltonian. Then one can show that finding a mean field equilibrium boils down to solving a system of a Hamilton-Jacobi-Bellman equation coupled with a Fokker-Planck-equation – or if one prefers the probabilistic approach – to a Forward-Backward Stochastic Differential Equation (FBSDE). The forward-backward structure of these differential equations is non-standard and a wide range of the literature regarding mean field games covers the analysis of these equations. For more details consider Bensoussan et al. (2013) and Carmona and Delarue (2018a, b) as well as the references therein. We remark, that in Lacker (2015) existence of mean field equilibria in mixed strategies is proven under continuity, measureability and boundedness conditions, which, in particular, do not include the assumption that a unique optimizer of the Hamiltonian exists. A similar existence result for mean field games with controlled jump-diffusion dynamics can be found in Benazzoli et al. (2018).

In Gomes et al. (2010, 2013) mean field games with finite state space have been introduced and thereafter several other (more general) mean field game models with finite state spaces haven been considered. We give an overview at the end of the introduction. Also applications of mean field games with finite state space have been considered. However, several applications consider finite action spaces (for example Kolokoltsov and Bensoussan (2016), Kolokoltsov and Malafeyev (2017), Guéant (2009a) and Besancenot and Dogguy (2015)) and for this type of models the literature covers only an existence result in mixed strategies (also called relaxed strategies) (see Cecchin and Fischer (2018)). Indeed, for non-trivial models with finite action spaces there are always population distributions for which more than one optimizer of the Hamiltonian exists, in which case most of the techniques presented in the literature so far are not applicable. For this reason for the previously mentioned examples the authors develop their own tools to solve their particular model: Kolokoltsov and Bensoussan(2016) and Kolokoltsov and Malafeyev(2017) only analyse stationary equilibria in deterministic strategies, Gueant(2009a) and Besancenot and Dogguy(2015) set up a dynamics equation only after analysing optimal decisions given a certain population distribution. They then also focus on stationary equilibria, as well as the dynamic behaviour close to these stationary equilibria and the effect of shocks.

In this paper, we will present general tools to compute stationary equilibria of mean field games with finite state and action space: We introduce the notion of stationary mean field equilibria into the mean field game model with finite state and finite action space presented in Doncel et al. (2016a). We remark that we consider a stationary equilibrium in an infinite horizon mean field game where the expected discounted reward ($\int_{0}^{\infty}e^{-\beta t}\ldots$) is maximized and not, as in many other settings, a stationary (ergodic) equilibrium, where the expected average reward ($\limsup_{T\rightarrow\infty}\int_{0}^{T}\ldots$) is maximized. Since the analytic formulation of Doncel et al. (2016a) is not suitable for this task, we formulate the model in a probabilistic way: The individual dynamics of a player is given by a time-inhomogeneous continuous time Markov chain with the generator depending on the current action and the current distribution of the population; the costs depend on the current state as well as the current action of the individual player and the current population distribution. As in Doncel et al. (2016a) in the case of dynamic equilibria, we only prove existence of stationary mean field equilibria in mixed strategies.

We focus on stationary equilibria for several reasons: First, searching for stationary equilibria reduces the complexity of the considered problem. More precisely, we can utilize the standard theory on Markov decision process with stationary transition rates and rewards and we can considered a fixed point problem in $\mathbb{R}^{S}$ instead of a fixed point problem in some function space. Second, the main focus of the economic models studied so far also lies in stationary equilibria. Third and linked to the last reason, one can often establish some kind of convergence/adjustment process towards stationary equilibria. More precisely, Gomes et al. (2013) prove that for mean field games with finite state space under certain assumptions every dynamic mean field equilibrium converges to the stationary (ergodic) equilibrium. In the case of continuous state spaces Cardaliaguet et al. (2012, 2013) proves that under certain assumptions every dynamic mean field equilibrium converges to the stationary (ergodic) equilibrium and Guéant (2009b) describes a cognitive process that converges if it is started close to stationary equilibrium indeed to the stationary (ergodic) equilibrium. Moreover, in the examples presented by Guéant (2009a) and Besancenot and Dogguy (2015) it is shown that there is local convergence of the trajectories of the dynamic equilibrium to the stationary (discounted reward) equilibrium.

Relying on our probabilistic formulation of the model, we will show existence of stationary equilibria under the same conditions as in the dynamic case. This is compared to Gomes et al. (2013) a surprising result as they need several additional assumptions compared to the dynamic existence result to establish existence of stationary equilibria.

Thereafter, we derive tools to compute all stationary equilibria (including those where the equilibrium strategy randomizes over different actions). As in standard mean field game models, we first have to solve an optimal control problem given a fixed flow of population distribution and second a fixed point problem, namely searching for flows of population distributions $m$ such that $m$ is the distribution of an individual player playing optimal given $m$. We will see that the first problem is equivalent to solving a standard Markov decision process with expected discounted reward criterion and we will show that the set of all randomized optimal stationary strategies is the convex hull of all deterministic optimal stationary strategies. For general dynamics, we cannot simplify the fixed point problem directly but we will provide a generally applicable reformulation of the necessary and sufficient balance equations inspired by the cut criterion for standard Markov chains, which often proves helpful in examples. Assuming irreducibility of the generators of the individual dynamics given any population distribution and any strategy, we can obtain all distributions of stationary equilibria as the fixed points of a set-valued map with convex values, which can be completely characterized by the transition rates and rewards given deterministic strategies.

1.1 Related Literature

As indicated previously we would like to sketch briefly the research regarding finite state mean field games: The starting point regarding the study of finite state mean field games were the models of Gomes et al. (2010) (in discrete time), Gomes et al. (2013) and Guéant (2011, 2015). We will focus on the continuous time models here: In both models, fully controllable transition rates are considered (Guéant (2011, 2015) additionally assumes that the players might not reach all other states from a given state) such that the player’s dynamics is given by a time-inhomogeneous Markov chain. The costs consist of instantaneous costs depending on the current state and action of the individual player as well as the current distribution of all players together with a terminal cost depending on the current state of the individual player and the current distribution of all players. In both models, assumptions are set up such that there is always a unique optimizer of the Hamiltonian. For both models then a system of forward-backward ODEs is presented, the solution of which yields a mean field equilibrium. Moreover, existence and uniqueness of solutions to these equations is discussed.

Guéant (2011) discusses further sufficient conditions for the existence of mean field equilibria (including a discrete state master equation); Gomes et al. (2013) studies stationary equilibria and establishes for contractive mean field games that a trend to equilibrium exists. Gomes et al. (2013) furthermore study an $N$-player game and the convergence of this game to the mean field game model. Additionally, as in the diffusion-based models, the class of potential mean field games is introduced, which has a simpler cost structure and, thus, allows for deeper results. Namely, the costs split in two additive terms, one term depending on the current state and the current population distribution, which is furthermore the gradient of a convex function, as well as one term depending on the current state and current action.

Several other authors discuss similar questions in models with more general individual dynamics, in particular the transition rates might not be fully controllable, but again assumptions were set which imply that there is a unique optimizer of the Hamiltonian: Carmona and Delarue (2018a, Chapter 7.2) provide a discussion of finite state mean field games, which is closely related to their exposition of standard mean field games models with continuous state space. They consider models in which transition rates depend on the current state, the current action and the population distribution and discuss existence and uniqueness results as well as a master equation. Basna et al. (2014) discusses mean field games were the dynamics are given by a non-linear Markov process with a generator that might additionally depend on the distribution of all other players and show under several assumptions that mean field equilibria are $\frac{1}{N}$-Nash equilibria for the corresponding $N$-player games.

Cecchin and Fischer (2018) present a mean field game with the individual dynamics given by stochastic differential equation driven by a stationary Poisson random measure, where again a dependence on the current population distribution is possible. They discuss existence (also in mixed strategies under mild continuity and boundedness assumptions) and uniqueness of mean field equilibria and furthermore show that mean field equilibria in open-loop and feedback strategies are $\epsilon_{N}$-Nash equilibria for the corresponding $N$-player game. Carmona and Wang (2018) provide a mean field game model with the dynamics given by a continuous time Markov chain with a generator which might depend on the current distributions of the states as well as the actions of all players. Using the semimartingale representation of continuous time Markov chains they again consider existence, uniqueness and the question when a mean field equilibrium is an approximate Nash equilibrium for the corresponding $N$-player game.

The model of Doncel et al. (2016a) (which we consider in this paper) does not require a unique optimizer of the Hamiltonian, but instead it is directly assumed that the action space is finite. The dynamics of each individual player are given by a differential equation specifying the transition rates, which in turn depend on the individual’s state and action as well as on the current population distribution. Existence of dynamic mean field equilibria is shown and a discrete time $N$-player game is considered, for which it is shown that mean field equilibria are $\epsilon$-Nash equilibria. Furthermore, the question of convergence is considered. More precisely, given a sequence of strategies $(\pi^{N})_{N\in\mathbb{N}}$ which are equilibria in the $N$-player game is there some sub-sequence converging to a mean field equilibrium? The answer to this question is positive if one considers local strategies (which only depend on the current state and time), but negative if one considers Markov strategies (which also depend on the current distribution of all players). The intuitive reason for this is that the “tit-for-tat”-principle cannot be applied in the limit (see Doncel et al. (2016b) for more details).

1.2 Organization of the Paper

The structure of the paper is as follows: Section 2 introduces the model in a probabilistic formulation. Section 3 discusses the individual control problem. In Section 4 we show that for all models fitting into our framework a stationary mean field equilibrium in mixed strategies exists. Section 5 first discusses the generally applicable cut criterion for our setting, then - given irreducibility of the generator - we propose a characterization of the set of all distributions of stationary mean field equilibria as the set of all fixed points of a suitable set-valued map which is completely determined by the dynamics and rewards given the deterministic strategies. Section 6 concludes the paper by showing how the presented tools can be applied to find stationary mean field equilibria in two examples.

2 The Model

Let $\mathcal{S}=\{1,\ldots,S\}$ ($S>1$) be the set of possible states of each player and let $\mathcal{A}=\{1,\ldots,A\}$ be the set of possible actions. With $\mathcal{P}(\mathcal{S})$ we denote the probability simplex over $\mathcal{S}$ and similarly for $\mathcal{P}(\mathcal{A})$. A (mixed) strategy is a measurable function $\pi:\mathcal{S}\times[0,\infty)\rightarrow\mathcal{P}(\mathcal{A})$, $(i,t)\mapsto(\pi_{ia}(t))_{a\in\mathcal{A}}$ with the interpretation that $\pi_{ia}(t)$ is the probability that at time $t$ and in state $i$ the player chooses action $a$. We say that a strategy $\pi=d:\mathcal{S}\times[0,\infty)\rightarrow\mathcal{P}(\mathcal{A})$ is deterministic if it satisfies for all $t\geq 0$ and for all $i\in\mathcal{S}$ that there is an $a\in\mathcal{A}$ such that $d_{ia}(t)=1$ and $d_{ia^{\prime}}=0$ for all $a^{\prime}\in\mathcal{A}\setminus\{a\}$. Throughout the presentation we often use the following equivalent representation, which is to represent a deterministic strategy as a function $d:\mathcal{S}\times[0,\infty)\rightarrow\mathcal{A},(i,t)\mapsto d_{i}(t)$ with the interpretation that $d_{i}(t)=a$ states that at time $t$ in state $i$ action $a$ is chosen. With $\Pi$ we denote the set of all (mixed) strategies and with $D$ the set of all deterministic strategies.

The individual dynamics of each player given a flow of population distributions $m:[0,\infty)\rightarrow\mathcal{P}(\mathcal{S})$ and a strategy $\pi:\mathcal{S}\times[0,\infty)\rightarrow\mathcal{P}(\mathcal{A})$ are given as a Markov process $X^{\pi}(m)$ on a given probability space $(\Omega,\mathcal{F},\mathbb{P})$ with given initial distribution $x_{0}$ and infinitesimal generator given by the $Q(t)$-matrix

[TABLE]

where for all $a\in\mathcal{A}$ and $m\in\mathcal{P}(\mathcal{S})$ the matrices $(Q_{\cdot\cdot a}(m))$ are conservative generators, that is $Q_{ija}(m)\geq 0$ for all $i,j\in\mathcal{S}$ with $i\neq j$ and $\sum_{j\in\mathcal{S}}Q_{ija}(m)=0$ for all $i\in\mathcal{S}$.

Given the initial condition $x_{0}\in\mathcal{P}(\mathcal{S})$, the goal of each player is to maximize his expected reward, which is given by

[TABLE]

where $x_{i}(t)$ is the probability that the individual player is in state $i$ at time $t$, $r:\mathcal{S}\times\mathcal{A}\times\mathcal{P}(\mathcal{S})\rightarrow\mathbb{R}$ is a real-valued function and $\beta\in(0,1)$ is the discount factor. That is, for a fixed population distribution $m:[0,\infty)\mapsto\mathcal{P}(\mathcal{S})$ we face a Markov decision process with expected discounted reward criterion and time-inhomogeneous reward functions and transition rates.

We will work under the following mild continuity assumption, which will ensure that there is indeed a Markov process with generator $Q^{\pi}$ (see Guo and Hernández-Lerma (2009, Appendix B+C) for details):

Assumption A1**.**

For all $i,j\in\mathcal{S}$ and all $a\in\mathcal{A}$ the function $m\mapsto Q_{ija}(m)$ mapping from $\mathcal{P}(\mathcal{S})$ to $\mathbb{R}$ is Lipschitz-continuous in $m$ . For all $i\in\mathcal{S}$ and all $a\in\mathcal{A}$ the function $m\mapsto r_{ia}(m)$ mapping from $\mathcal{P}(\mathcal{S})$ to $\mathbb{R}$ is continuous in $m$.

With these preparations, we can introduce the concept of dynamic mean field equilibria:

Definition 2.1**.**

Given an initial distribution $m_{0}$, a mean field equilibrium is a flow of population distributions $m:[0,\infty)\rightarrow\mathcal{P}(\mathcal{S})$ with $m(0)=m_{0}$ and a strategy $\pi:\mathcal{S}\times[0,\infty)\rightarrow\mathcal{P}(\mathcal{A})$ such that

• •

the distribution of the process $X^{\pi}(m)$ at time $t$ is given by $m(t)$

• •

$V{{}_{m_{0}}}(\pi,m)\geq V{{}_{m_{0}}}(\pi^{\prime},m)$ for all $\pi^{\prime}\in\Pi$ .

As in standard game theory, our concept of mean field equilibrium captures the intuitive idea that no player wants to deviate: Given that all players play according to strategy $\pi$ the population’s distribution will be $m$. If an individual player now evaluates whether he wants to deviate from playing $\pi$ he asks whether there is a strategy that yields a higher payoff given $m$. Due to the second condition this is not possible. Therefore, we indeed face an equilibrium in the standard economic sense.

Remark 2.1*.*

Using the Kolmogorov forward equation (Guo and Hernández-Lerma, 2009, Proposition C.4), we see that the first condition implies the analytic condition used in Doncel et al. (2016a) to characterize mean field equilibria, which states that $m$ is solution of

[TABLE]

with initial condition $m(0)=m^{0}$.

Remark 2.2*.*

The definition of strategies we adopt here is unusual in classical game theory. However, in the setting of mean field games it is sensible. Indeed, given the initial state of the system and the strategy of the other players the behaviour of the system is fully determined. Thus, it suffices for the individual agent to know the initial global state (see Caines et al. (2017)).

In order to define stationary mean field equilibria, we first introduce the notion of stationary strategies: A stationary strategy is a map $\pi:\mathcal{S}\times[0,\infty)\rightarrow\mathcal{P}(\mathcal{A})$ such that $\pi_{ia}(t)=\pi_{ia}$ for all $t\geq 0$. Again we denote by $\Pi^{s}$ the set of all stationary strategies and by $D^{s}$ the set of all deterministic stationary strategies.

Definition 2.2**.**

A stationary mean field equilibrium is given by a stationary strategy $\pi$ and a vector $m\in\mathcal{P}(\mathcal{S})$ such that

• •

the law of $X^{\pi}(m)$ at any point in time $t$ is given by $m$

• •

for any initial distribution $x_{0}\in\mathcal{P}(\mathcal{S})$ we have $V{{}_{x_{0}}}(\pi,m)\geq V{{}_{x_{0}}}(\pi^{\prime},m)$ for all $\pi^{\prime}\in\Pi$.

This notion is a sensible formalization of stationary equilibria: Given the strategy $\pi$ the population’s distribution will be $m$ for all time points. An individual agent at a given time point can be in any state, however, if he evaluates whether he wants to deviate from playing $\pi$, the second condition ensures that this is not beneficial for him. Thus, he has no incentive to deviate from the equilibrium strategy $\pi$, which means that the population will indeed remain in the stationary equilibrium regime of playing $\pi$.

Remark 2.3*.*

We remark that the matrix $Q_{ij}^{\pi}(m,t)$ does not depend on $t$ in this context, therefore, we write $Q^{\pi}_{ij}(m):=Q^{\pi}_{ij}(m,t)$. Using this, we obtain that the first condition is equivalent to

[TABLE]

Moreover, we remark that the second condition requires $\pi$ to be optimal among all strategies, not only those that are stationary.

Remark 2.4*.*

In contrast to the standard models where the assumptions usually imply that a unique optimal best response exists, the mean field equilibria we consider are not fully specified by the distribution, as it might happen that several actions are simultaneously optimal and induce the same distribution. However, the dynamic mean field equilibrium is fully specified by describing the equilibrium strategy, as one can show using standard techniques (Walter, 1998, Theorem 10.XX) that there is at most one solution to the differential equation. For the stationary mean field equilibrium, this again does not hold true, as it might happen that given a strategy there are multiple stationary distributions. For this reason, we define mean field equilibria always as pairs of the equilibrium distribution and the equilibrium strategy.

Remark 2.5*.*

We remark that for non-trivial models (in the sense that there is not one action that maximizes the Hamiltonian for every population distribution) we always obtain population distributions at which several actions maximize the Hamiltonian: Since $Q(\cdot)$ and $r(\cdot)$ are continuous, also the Hamiltonian is continuous in $m$. Therefore, if we fix the costate variables, the sets of population distributions in which a particular action is a maximizer of the Hamiltonian are closed. Since the action space is finite and the set of all population distribution vectors is connected, we obtain that if there is more than one action that maximizes the Hamiltonian for some population distribution, then the set of population distributions where several actions simultaneously maximize the Hamiltonian is non-empty. This implies that for the case of finite action spaces the assumption that a unique maximizer of the Hamiltonian exists is violated in all interesting cases. Thus, new methods for the analysis of these models are necessary.

3 The Individual Control Problem

This section is devoted to the analysis of the individual control problem: We propose a simple approach to determine which strategies are optimal for a given population distribution and show that optimal stationary strategies are convex combinations of particular deterministic stationary strategies.

We start by showing that given a stationary population distribution $m\in\mathcal{P}(\mathcal{S})$ the individual player’s control problem is equivalent to a continuous time Markov decision process with expected discounted reward criterion (see Guo and Hernández-Lerma (2009) for a definition and general results).

Lemma 3.1**.**

Let $m\in\mathcal{P}(\mathcal{S})$ be a population distribution. A Markovian randomized strategy $\pi$ is optimal in our model given $m$, i.e. achieves for all $x_{0}\in\mathcal{P}(\mathcal{S})$ the maximum value of $V{{}_{x_{0}}}(\pi^{\prime},m)$ among all strategies $\pi^{\prime}\in\Pi$, if and only if it is discounted reward optimal for the continuous time Markov decision process with transition rates $Q_{ija}(m)$, rewards $r_{ia}(m)$ and discount factor $\beta$. In particular, there is a stationary strategy $\pi\in\Pi^{s}$ that satisfies $V_{x_{0}}(\pi,m)\geq V_{x_{0}}(\pi^{\prime},m)$ for all $x_{0}\in\mathcal{P}(\mathcal{S})$ and all $\pi^{\prime}\in\Pi$.

Proof.

Assumption A1 ensures that the value function is finite for every population distribution function and every individual strategy since $r_{ia}(\cdot)$ is, as a continuous function on a compact space, uniformly bounded. Thus, we can rewrite the value function by using the representation $x_{i}(t)=\sum_{k\in\mathcal{S}}x_{k}^{0}\cdot p^{\pi^{0}}(0,k,t,i)$:

[TABLE]

where $V_{k}^{\pi^{0}}(m)$ is the expected discounted reward of a continuous time Markov decision process with expected discounted reward criterion with the above-mentioned rates and rewards when the initial state is $k$. Since a strategy $\pi$ is optimal for the continuous time Markov decision process if it maximizes all $V_{k}^{\pi}(m)$ simultaneously, we obtain the desired equivalence. The last statement directly follows for the classical theory for Markov decision process, see for example Guo and Hernández-Lerma (2009, Chapter 4). ∎

Now, we show that the set of all optimal stationary strategies is the convex hull of all deterministic optimal stationary strategies.

Theorem 3.2**.**

Let $m\in\mathcal{P}(\mathcal{S})$. Write

[TABLE]

with

[TABLE]

where $V^{\ast}(m)$ is the value function of the associated Markov decision process. Then $\mathcal{D}(m)$ is non-empty. Furthermore, a stationary strategy is optimal for our model given $m$ if and only if it is a convex combination of strategies from $\mathcal{D}(m)$.

Proof.

By the previous lemma we know that the individual’s control problem is equivalent to the continuous time Markov decision process with expected discounted reward criterion with discount factor $\beta$, transition rates $Q_{ija}(m)$ and reward function $r_{ia}(m)$. Since we consider a finite state space, we obtain, using the uniformization procedure (Guo and Hernández-Lerma (2009, Remark 6.1), Kakumanu (1977)), an equivalent discrete time Markov decision process with expected discounted reward criterion. Its discount factor is $\alpha=\frac{||Q(m)||}{\beta+||Q(m)||}$. Writing $||Q(m)||=\sup_{i\in\mathcal{S},a\in\mathcal{A}}-Q_{iia}(m)$, the transition rates are given by

[TABLE]

and the reward functions are given by

[TABLE]

Simple computations yield that

[TABLE]

with $\tilde{V}^{\ast}_{j}(m)$ being the value function of the discrete time Markov decision process, which is indeed equal to the value function of the continuous time Markov decision process (see Kakumanu (1977) for details; note however, that he does not adjust the rewards, which yields the proportional factor for the value functions in his setting).

Now we can prove the statement for discrete time Markov decision processes relying on the rich theory developed for these problems (see Puterman (1994)): We first note that the set $\mathcal{D}(m)$ in non-empty since $\mathcal{A}$ is finite. Moreover, by Puterman (1994, Corollary 6.2.8), any deterministic strategy in our set $\mathcal{D}(m)$ is indeed optimal. Enumerate $\mathcal{D}(m)=\{d_{1},\ldots,d_{n}\}$ and let $\pi\in\Pi^{s}$ be a convex combination of strategies in $\mathcal{D}(m)$, that is

[TABLE]

By Puterman (1994, Theorem 6.1.1) the reward function given a certain stationary strategy can be written as the unique solution of $v=\bar{r}^{\pi}(m)+\alpha\bar{P}^{\pi}(m)v$ with

[TABLE]

Noting that $\bar{r}^{\pi}$ and $\bar{P}^{\pi}$ are linear in $\pi$, we can rewrite the policy evaluation equation as

[TABLE]

Since for all $i\in\{1,\ldots,n\}$ the deterministic stationary strategy $d_{i}$ is optimal it follows that $V^{\pi}=V^{\ast}$ is the unique solution of the policy evaluation equation:

[TABLE]

By Puterman (1994, Theorem 6.2.2 and Theorem 6.2.5), which states that in our setting the unique solution of the optimality equation is $V^{\ast}$, and by Puterman (1994, Theorem 6.2.6), which states that a strategy is optimal if and only if its value function is a solution of the optimality equation, we obtain that the strategy $\pi$ is optimal.

To show the converse implication we assume that $\pi$ is not a convex combination of deterministic strategies from $\mathcal{D}(m)$. One easily sees that there is still a representation of $\pi$ as a convex combination of arbitrary deterministic strategies by considering

[TABLE]

Moreover, any convex combination of deterministic strategies representing the strategy $\pi$ has a summand $d\notin\mathcal{D}(m)$ with positive weight. This means that for the strategy $d$ there is a state $i\in\mathcal{S}$ and an action $\tilde{a}\in\mathcal{A}\setminus O_{i}(m)$ such that $d(i)=\tilde{a}$. This implies that also the stationary strategy $\pi$ chooses that action $\tilde{a}$ in state $i$ with positive probability, that is $\pi_{i\tilde{a}}>0$. This means for the $i$-th component of the strategy’s expected discounted reward:

[TABLE]

Note that the second lines follows from $V^{\pi}\leq V^{\ast}$ and the third line follows from the fact that $\tilde{a}\notin O_{i}$ is chosen with positive probability $\pi_{i\tilde{a}}$.

As now $V^{\pi}(i)<V^{\ast}(i)$ and by the finiteness of $\mathcal{S}$ and $\mathcal{A}$ an optimal strategy achieving value $V^{\ast}$ exists, it follows that $\pi$ is not optimal. ∎

We note that we reduced the problem of determining which of the infinitely many strategies are indeed optimal for a given $m\in\mathcal{P}(\mathcal{S})$ to the problem of determining which of the finitely many deterministic strategies are optimal.

This theorem yields a basic guideline for finding all mean field equilibria: For each point $m\in\mathcal{P}(\mathcal{S})$ determine the set $\mathcal{D}(m)$ of all optimal strategies. Since there are only finitely many deterministic stationary strategies, this yields to a partition of $\mathcal{P}(\mathcal{S})$. Let us write $\text{Opt}(A_{1}\times\ldots\times A_{S})$ for the set of all $m\in\mathcal{P}(\mathcal{S})$ such that $d\in\mathcal{D}(m)$ if and only if $d_{i}\in A_{i}$ for all $i\in\mathcal{S}$. Thus, we then have to search in each of the sets $\text{Opt}(A_{1}\times\ldots\times A_{S})$ for fixed points of the dynamics given all those deterministic stationary strategies satisfying $d(i)\in A_{i}$ for all $i\in\mathcal{S}$.

Furthermore, the results allows us to prove that a game that is not trivial in the sense that is there are two different population distributions such that different actions are optimal for each of them, has a closed, non-empty set of points where infinitely many strategies are optimal. Thus, we indeed have to consider infinitely many (potentially different) fixed point problems in order to compute all stationary mean field equilibria. Indeed, by the classical theory of Markov decision processes we know that those deterministic strategies are optimal that maximize the expected discounted reward $V^{d}(m)$. Noting that the expected discounted reward is given by

[TABLE]

and that $r^{d}(\cdot)$ and $Q^{d}(\cdot)$ are continuous, it follows that also $V^{d}(\cdot)$ is continuous. The set of all those points where a certain strategy $d$ is optimal is the preimage of $[0,\infty)^{S}$ under the continuous map $V^{d}(m)-\max_{d^{\prime}\in D^{s}}V^{d^{\prime}}(m)$ and thus closed. Whenever there are two strategies that are optimal for distinct population distributions we have two (or more) non-empty closed sets of points $m\in\mathcal{P}(\mathcal{S})$ for which a certain deterministic strategy is one (but possibly not the only) optimal strategy. Since $\mathcal{P}(\mathcal{S})$ is itself closed and connected, there will be a non-empty, closed set for which at least two deterministic stationary strategies, and thus infinitely many stationary strategies are optimal.

4 Existence

In Section 3 we proved that there is a stationary strategy that is optimal among all (also time-dependent) strategies. Moreover, we proved that a stationary strategy is optimal for $m\in\mathcal{P}(\mathcal{S})$ if and only if it is a convex combination from $\mathcal{D}(m)$, which is the set of all deterministic stationary strategies that are optimal.

Using this, we will prove that whenever the assumption A1 holds there exists a stationary mean field equilibrium. We will adapt the ideas presented in Doncel et al. (2016a) to prove this. More precisely, we will show the existence of a fixed point of an associated best response map in the dynamics. This map maps to each point $m$ all the stationary points of $Q^{\pi}(m)$ given that $\pi$ is an optimal strategy for $m$. In contrast to the proof of the existence of dynamic equilibria presented in Doncel et al. (2016a) we do not only rely on standard calculus arguments, but instead the proof crucially relies on our probabilistic representation of the problem.

We define the best response map $\phi:\mathcal{P}(\mathcal{S})\rightarrow 2^{\mathcal{P}(\mathcal{S})}$, where $2^{\mathcal{P}(\mathcal{S})}$ denotes the power set of $\mathcal{P}(\mathcal{S})$, by setting

[TABLE]

We will now show using Kakutani’s fixed point theorem that this map has a fixed point and that each fixed point of this map induces a stationary mean field equilibrium:

Theorem 4.1**.**

Given assumption A1 there is a stationary mean field equilibrium.

Proof.

We show that $\phi$ has a fixed point using Kakutani’s fixed point theorem (Border, 1985, Corollary 15.3), since any such fixed point yields to a stationary mean field equilibrium: Indeed, for any fixed point $m$ we find a strategy $\pi\in\mathcal{D}(m)$ such that $0=m^{T}Q^{\pi}(m)$. Since by Lemma 3.1 and Theorem 3.2 we moreover have that $V_{x_{0}}(\pi,m)\geq V_{x_{0}}(\pi^{\prime},m)$ for all $x_{0}\in\mathcal{P}(\mathcal{S})$ and all $\pi^{\prime}\in\Pi$ the pair $(m,\pi)$ constitutes a stationary mean field equilibrium.

We first note that $\phi(m)$ is non-empty for all $m\in\mathcal{P}(\mathcal{S})$. By Theorem 3.2 the set $\mathcal{D}(m)$ is non-empty. Since any continuous time Markov chain with finite state space has at least one stationary distribution there exists an $x\in\mathcal{P}(\mathcal{S})$ such that $0=x^{T}Q^{\pi}(m)$, which yields that $x\in\phi(m)$.

Furthermore, for each $m\in\mathcal{P}(\mathcal{S})$ the set $\phi(m)$ is convex: Let $x^{1},x^{2}\in\phi(m)$ be two distinct points. Then, by definition of $\phi$, we find two strategies $\pi^{1},\pi^{2}\in\text{conv}(\mathcal{D}(m))$ such that

[TABLE]

Define $z_{ia}^{1}:=x_{i}^{1}\pi_{ia}^{1}$ and $z_{ia}^{2}=x_{i}^{2}\pi_{ia}^{2}$, which satisfy

[TABLE]

Now let $\alpha\in[0,1]$ be arbitrary and define $x^{3}=\alpha x^{1}+(1-\alpha)x^{2}$. Then $z^{3}=\alpha z^{1}+(1-\alpha)z^{2}$ satisfies

[TABLE]

which means that

[TABLE]

satisfies $0=(x^{3})^{T}Q^{\pi^{3}}(m)$. It remains to verify that $\pi^{3}\in\text{conv}(\mathcal{D}(m))$. For this we note that $\pi_{ia}^{3}>0$ of and only if $z_{ia}^{3}>0$, which in turn is equivalent to the requirement that $z_{ia}^{1}>0$ or $z_{ia}^{2}>0$. This can only happen if $\pi_{ia}^{1}>0$ or $\pi_{ia}^{2}>0$. Thus, since $\pi^{1},\pi^{2}\in\text{conv}(\mathcal{D}(m))$, also $\pi^{3}\in\text{conv}(\mathcal{D}(m))$.

We now verify that $\phi$ has a closed graph, that is, that for any sequence $(m^{n})_{n\in\mathbb{N}}\in\mathcal{P}(\mathcal{S})^{\mathbb{N}}$ and $x^{n}\in\phi(m^{n})$ for all $n\in\mathbb{N}$ with $\lim_{n\rightarrow\infty}m^{n}=m$ and $\lim_{n\rightarrow\infty}x^{n}=x$ we indeed have $x\in\phi(m)$: Let $(m^{n},x^{n})_{n\in\mathbb{N}}$ be a converging sequence satisfying $x^{n}\in\phi(m^{n})$ for all $n\in\mathbb{N}$. We denote its limit by $(m,x)$. By definition of $\phi$ we find a sequence $\pi^{n}\in\text{conv}(\mathcal{D}(m^{n}))$ such that $0=x^{n}Q^{\pi^{n}}(m^{n})$. By compactness of $\Pi^{s}$ we find a converging subsequence $(\pi^{n_{k}})_{k\in\mathbb{N}}$ with limit $\pi\in\Pi^{s}$. For any $k\in\mathbb{N}$ let $A_{1}^{k}\times\ldots\times A_{S}^{k}\subseteq\mathcal{A}^{S}$ be such that $\pi_{ia}^{n_{k}}>0$ for all $i\in\mathcal{S},a\in A_{i}^{k}$ and $\pi_{ia}=0$ for all $i\in\mathcal{S},a\notin A_{i}^{k}$. Since $\mathcal{A}^{S}$ is finite we find a set $A_{1}\times\ldots\times A_{S}$ that occurs infinitely often in the sequence $(A_{1}^{k}\times\ldots\times A_{S}^{k})_{k\in\mathbb{N}}$. From this we obtain that $\pi_{ia}=0$ for all $i\in\mathcal{S}$ and $a\notin A_{i}$. Moreover, since $\pi^{n}\in\text{conv}(\mathcal{D}(m^{n}))$ we obtain that for all $l\in\mathbb{N}$ such that $A_{1}\times\ldots\times A_{S}=A_{1}^{l}\times\ldots\times A_{s}^{l}$ we have $V^{d}(m^{n_{l}})=V^{\ast}(m^{n_{l}})$ for all deterministic strategies satisfying $d(i)\in A_{i}$ for all $i\in\mathcal{S}$. By continuity of $V^{d}(\cdot)$ and $V^{\ast}(\cdot)$, we obtain that for all these strategies $V^{d}(m)=V^{\ast}(m)$. Thus, $\pi\in\text{conv}(\mathcal{D}(m))$. Furthermore, by continuity of $Q(\cdot)$, we obtain that

[TABLE]

which shows that $x\in\phi(m)$.

Using that $\mathcal{P}(\mathcal{A})$ is a compact metric space and that the graph of $\phi$ is closed, we obtain that the values $\phi(m)$ are compact as the limit of any sequence $x^{n}\in\phi(m)$ lies in $\phi(m)$.

Now Kakutani’s fixed point theorem (Border, 1985, Corollary 15.3) yields a fixed point $m\in\phi(m)$, which proves the desired claim. ∎

5 The Fixed Point Problem

5.1 The Cut Criterion

To solve the fixed point problem we have to determine the solutions of the equation $m^{T}Q^{\pi}(m)=0$ for all strategies $\pi$ that are optimal for some population distribution and then we have to check whether $\pi$ is indeed optimal for these solutions. In many settings this task is not simple (see Section 6.2). However, often a cut criterion similar to the one used for Markov chains is useful, although it is just a reformulation of the balance equation $m^{T}Q^{\pi}(m)=0$ (see Kelly (1979, Lemma 1.4) for a description of the criterion for standard continuous time Markov chains). The criterion states that if we partition the state space of the Markov chain into two sets, then the probability flow from one set to the other has to equal the probability flow from this other set to the first.

The particular use of the criterion is that in most models that have been consider so far there has always been a set of states for which the dynamics to and from this set cannot be influenced by the player by choosing a particular strategy. This means that any mean field equilibrium irrespective of the chosen strategy has to satisfy certain equations coming from the cut criterion, which could be obtained from the standard balance equations $Q^{\pi}(m)m=0$ only by sensible rearrangements. In Section 6.2 we will show that the criterion indeed simplifies the search for fixed points.

Theorem 5.1**.**

Let $\pi$ be a stationary strategy and let $\mathcal{T}\subseteq\mathcal{S}$. Then any stationary population distribution satisfies

[TABLE]

Proof.

The stationarity condition reads for all $j\in\mathcal{S}$

[TABLE]

furthermore, since $Q^{\pi}(m)$ is conservative, we have for all $j\in\mathcal{S}$

[TABLE]

This yields for all $j\in\mathcal{S}$

[TABLE]

Summing this identity over all $j\in\mathcal{T}$ yields

[TABLE]

Subtracting the identity

[TABLE]

yields the desired result. ∎

5.2 Mean Field Equilibria are Fixed Points of a Specific Map

This section is devoted to proving an explicit characterization of $\phi(m)$, which has been introduced in Section 4, in terms of the deterministic maps $x^{d}(\cdot)$ for those strategies $d$ that are optimal for $m$. In order to show this we will need irreducibility of $Q^{\pi}(m)$ for all strategies $\pi\in\Pi^{s}$. Note that it is sufficient to verify irreducibility for all deterministic strategies $d\in D^{s}$ since any stationary strategy $\pi$ is a convex combination of deterministic strategies and thus $Q^{\pi}=\sum_{d\in D^{s}}\lambda_{d}Q^{d}$ is also irreducible.

The main consequence of $Q^{\pi}(m)$ being irreducible is that there is a unique stationary distribution of the continuous time Markov chain (CTMC) with generator $Q^{\pi}(m)$ (see Durrett (1999, Corollary I.4.6 and Theorem I.4.7), Norris (1997, Theorem 3.5.1)). This observation allows us to formulate the main theorem, which we will prove in the rest of the section:

Theorem 5.2**.**

Let $m\in\mathcal{P}(\mathcal{S})$ such that $Q^{d}(m)$ is irreducible for all $d\in D^{s}$. Let, furthermore, $\mathcal{D}(m)=\{d_{1},\ldots,d_{n}\}$ be the set of all deterministic optimal strategies for $m$. Then

[TABLE]

with $x=x^{d_{k}}(m)$ being the unique solution of $0=\sum_{i\in\mathcal{S}}\sum_{a\in\mathcal{A}}x_{i}Q_{ija}(m)d_{ia}^{k}$.

The proof of this theorem basically relies on the idea to characterize the stationary distribution $x^{\pi}(m)$ of the CTMC with irreducible generator $Q^{\pi}(m)$ by a closed form expression and to show thereafter that $x^{\pi}(m)$ is a convex combination of $(x^{d}(m))_{d\in D^{s}}$. In order to follow this programme we have to prove several properties of the generator matrix $Q$ starting with the following lemma regarding the structural properties of an irreducible, conservative generator $Q\in\mathbb{R}^{S\times S}$, more precisely regarding the minor $Q_{SS}^{\prime}$, which arises from $Q$ by deleting the last row and column:

Lemma 5.3**.**

Let $Q\in\mathbb{R}^{S\times S}$ be an irreducible, conservative generator matrix. Then all eigenvalues of the minor $Q_{SS}^{\prime}$ have negative real part. Consequently $Q_{SS}^{\prime}$ has full rank and

[TABLE]

The technical proof can be found in the appendix.

Noting the last column is the negative sum of all other columns we obtain the following corollary:

Corollary 5.4**.**

Let $Q\in\mathbb{R}^{S\times S}$ be an irreducible, conservative generator matrix. Then the rank of $Q$ is $S-1$.

With these two results, we can now explicitly characterize the stationary distribution given a stationary strategy $\pi$ and a population distribution $m$:

Lemma 5.5**.**

Let $\pi\in\Pi^{s}$ be a stationary strategy and $m\in\mathcal{P}(\mathcal{S})$ such that $Q^{\pi}(m)$ is irreducible. Let $\tilde{Q}^{\pi}(m)$ be the transpose $(Q^{\pi}(m))^{T}$ where the last row is replaced by $(1,\ldots,1)$. Then we have that the unique stationary distribution $x^{\pi}(m)$ is given by

[TABLE]

Furthermore, we can write

[TABLE]

with

[TABLE]

Proof.

The stationary distribution of our process is uniquely determined by (Asmussen, 2003, Theorem II.4.2)

[TABLE]

Since the last equation of the system $0=x^{\pi}(m)^{T}Q^{\pi}(m)$ is the negative sum of all other equations, we obtain that the system (3) is equivalent to

[TABLE]

which by definition of $\tilde{Q}^{\pi}$ is $(0,\ldots,0,1)^{T}=\tilde{Q}^{\pi}(m)x^{\pi}(m)$.

We now show that the rank of the matrix $\tilde{Q}^{\pi}(m)$ is $S$, as in this case we can invert the matrix. As in Resnick (1992, p.137-139) we rely on the existence of the stationary distribution given $Q^{\pi}(m)$. In order to show that $\tilde{Q}^{\pi}(m)$ has full rank, we show that $y^{T}\tilde{Q}^{\pi}(m)=0$ implies that $y=0$. For the stationary distribution $x^{\pi}(m)$ given $Q^{\pi}(m)$ we have

[TABLE]

Thus,

[TABLE]

implies that

[TABLE]

Since by Lemma 5.3 the matrix $(Q^{\pi}(m))_{SS}^{\prime}$ has full rank we obtain that $y_{1}=\ldots=y_{S-1}=0$, which proves that $\tilde{Q}^{\pi}(m)$ has full rank.

The last part of the statement simply follows from Cramer’s rule together with the Laplace expansion of $\det(\tilde{Q}^{\pi}(m))$ along the last line

[TABLE]

and the observation that

[TABLE]

as $\tilde{Q}^{\pi}(m)$ differs from $Q^{\pi}(m)^{T}$ only in the $S$-th row. ∎

In order to establish the desired result on characterizing the convex set $\phi(m)$ solely in terms of transition rates and rewards for deterministic strategies, one final preparation has to be made: We have to show that the determinant of $\tilde{Q}^{\pi}(m)$ has uniform sign over all $\pi\in\Pi^{s}$: For this write $d^{(a_{1},\ldots,a_{S})}$ for the deterministic strategy satisfying $d(i)=a_{i}$. Then it holds that

[TABLE]

Now a simple application of the intermediate value theorem yields the desired result:

Lemma 5.6**.**

Let $m\in\mathcal{P}(\mathcal{S})$ be a population distribution such that $Q^{d}(m)$ is irreducible for all $d\in D^{s}$. Then $\det(\tilde{Q}^{\pi}(m))$ has uniform sign over $\pi\in\Pi^{s}$.

Proof.

We note that the map $\pi\mapsto\tilde{Q}^{\pi}(m)$, which ranges from $\Pi^{s}$ to $\mathbb{R}^{S\times S}$ is continuous. Since the determinant is also a continuous function, we see that $\pi\mapsto\det(\tilde{Q}^{\pi}(m))$ is a continuous function. By Lemma 5.5 we have that $\det(\tilde{Q}^{\pi}(m))\neq 0$ for all $\pi\in\Pi^{s}$. If there would be a strategy $\pi_{1}$ and a strategy $\pi_{2}$ such that $\det(\tilde{Q}^{\pi_{1}}(m))<0$ and $\det(\tilde{Q}^{\pi_{2}}(m))>0$, then, by the intermediate value theorem, there would be a $\pi\in\Pi^{s}$ such that $\det(\tilde{Q}^{\pi}(m))=0$, which would be a contradiction. ∎

With all these preparations, we can now prove the characterization result stated in the beginning of the section:

Proof of Theorem 5.2.

For readability we suppress the dependence of $Q$ and $x$ on $m$.

Let $\pi=\sum_{(a_{1},\ldots,a_{n})\in\mathcal{A}^{S}}\lambda_{(a_{1},\ldots,a_{S})}d^{(a_{1},\ldots,a_{S})}$. Note that $\lambda_{(a_{1},\ldots,a_{S})}$ is zero for all non-optimal strategies by Theorem 3.2. With $\hat{Q}$ being the matrix $Q^{T}$ without the last row, we can now write $\tilde{Q}^{\pi}$ as follows:

[TABLE]

As the determinant is linear in columns and we have $\sum_{a_{i}\in\mathcal{A}}\pi_{ia_{i}}=1$ for all $i\in\mathcal{S}$ we obtain

[TABLE]

Similarly, we obtain using that

[TABLE]

holds for all $a_{1},\ldots,a_{S},\tilde{a}_{i}\in\mathcal{A}$, that

[TABLE]

for all $\tilde{a}_{i}\in\mathcal{A}$. This implies

[TABLE]

Thus, we see that $x^{\pi}$ is a linear combination of $x^{d^{(a_{1},\ldots,a_{S})}}$ for any stationary strategy $\pi$ and the coefficients are given by

[TABLE]

From (5.2) we obtain that

[TABLE]

By Lemma 5.6 we note that the signs of the determinants $\tilde{Q}^{\pi}$ and $\tilde{Q}^{\pi^{\prime}}$ are the same for any two strategies $\pi,\pi^{\prime}$. Thus, as $\pi_{ia}\geq 0$ for all $i\in\mathcal{S},a\in\mathcal{A}$ we obtain that $\lambda_{(a_{1},\ldots,a_{S})}\geq 0$ for all $(a_{1},\ldots,a_{S})\in\mathcal{A}^{S}$. To conclude, writing $\mathcal{D}(m)=\{d^{1},\ldots,d^{n}\}$, every point in $\phi(m)$ is a convex combination of $x^{d_{1}},\ldots,x^{d_{n}}$. Moreover, any convex combination of $x^{d^{1}},\ldots,x^{d^{n}}$ is the stationary point given a strategy $\pi\in\text{conv}(\mathcal{D}(m))$, which by Theorem 3.2 yiels that these points lie in $\phi(m)$. ∎

Thus, in order to find all mean field equilibria it is sufficient to follow the following programme: First, compute for all sets $A_{1}\times\ldots,\times A_{S}\subseteq\mathcal{A}^{S}$ the set $\text{Opt}(A_{1}\times\ldots A_{S})$, which collects all those points $m\in\mathcal{P}(\mathcal{S})$ for which $\mathcal{D}(m)=\{d\in D^{S}:d(i)\in A_{i}\}$. Second, find all fixed points of the map $(\text{conv}\{x^{(a_{1},\ldots,a_{S})}(\cdot):(a_{1},\ldots,a_{S})\in A_{1}\times\ldots\times A_{S}\}$, that lie in $\text{Opt}(A_{1}\times\ldots A_{S})$. Writing $FP(f)$ for the set of all fixed points of the map $f$, we obtain the following result:

Theorem 5.7**.**

Assume that there is a set $\mathcal{T}\subseteq\mathcal{P}(\mathcal{S})$ such that for all $m\in\mathcal{T}$ and all $\pi\in\Pi^{s}$ the matrix $Q^{\pi}(m)$ is irreducible. Then the set of all distributions lying in $\mathcal{T}$ induced by some stationary mean field equilibrium is given by

[TABLE]

In case of constant dynamics, that is $Q_{ija}(m)=Q_{ija}$ for all $m\in\mathcal{P}(\mathcal{S})$ the second step of this programme is simple, since the maps $x^{d}(\cdot)$ are constant with value $(\tilde{Q}^{d})^{-1}\cdot(0,\ldots,0,1)^{T}$ and this implies that the unique fixed point is $(\tilde{Q}^{d})^{-1}\cdot(0,\ldots,0,1)^{T}$. Thus, we can characterize the set of all mean field equilibria as explicitly by only computing the optimality sets and the stationary points given $Q^{d}$ as follows:

Corollary 5.8**.**

Let the dynamics be constant, that is $Q_{ija}(m)=Q_{ija}$ for all $m\in\mathcal{P}(\mathcal{S})$, and let the generators $Q^{d}$ given any strategy $d$ be irreducible. Then the set of all distributions induced by some stationary MFE is given by

[TABLE]

6 Examples

This section presents two mean field game models, which have been considered in similar versions in the literature before, and illustrates the application of the techniques presented before. For both examples we will first solve the individual control problem given a fixed population distribution and compute given this the non-empty optimality sets $\text{Opt}(A_{1}\times\ldots A_{S})$. Thereafter we will solve the fixed point problems relyin on the approaches introduced before and sketch how the full characterization of the equilibria can be obtained.

6.1 A Consumer Choice Model

The model we present now is a model with constant dynamics, that is, the dynamics do not depend on the current population distribution. The context and utility function are similar to a model introduced by Gomes et al. (2014) as a toy example on which the authors demonstrated numerical methods for a specific class of finite state mean field game models, namely those yielding systems of hyperbolic partial differential equations. Note however, that the action spaces and choice options of the players differ systematically.

The model is inspired by consumer choices in the mobile phone sector. The utility of using a certain provider is increasing in the share of customers using it, whereas the costs are constant. We assume that the utility coming from the other customers in service is given by the isoelastic utility function $\ln(m_{i})+s_{i}$, with $s_{i}\in\mathbb{R}$. Note that since $\ln(m_{i})$ is always negative for our choice of $m_{i}$, one cannot interpret $s_{i}$ as costs directly, but one rather has to think of $s_{i}$ consisting of two components: the costs themselves and some base utility from service provision. The players can now choose in our model whether to stick to their provider $i$ or whether to switch to the other provider, in this case the player additionally faces a time-unit switching cost $c_{i}\geq 0$. Note that for technical reasons it is important that the player always faces independent of the chosen strategy a small risk of going to the other provider.

These choice options differ substantially from the model in Gomes et al. (2014), where the players could continuously control the rates at which they switch to the other state and were facing costs corresponding to the square of the rate. From an applied point of view it is questionable, in particular when agents are not experts in the game at hand, that players indeed understand what it means to control the transition rates of a Markov chain. Indeed, economic experiments show that most people cannot understand the true effect of random devices even in simple settings (Walker and Wooders, 2008).

The formal description of the model is now given as follows: For technical reasons (i.e. to define the rewards for the case $m_{i}=0$ properly) we introduce for small enough $\delta>0$ the function $f_{\delta}:\mathbb{R}\rightarrow\mathbb{R}$ given by

[TABLE]

which is increasing on $[0,1]$. Using this we then define the transition rates and reward functions by

[TABLE]

where $0<\epsilon<b$ and $s_{1},s_{2},c>0$.

In order to analyse the model we first solve the optimality equations

[TABLE]

which directly yield that it is optimal to change in state $1$ if and only if $V_{1}(m)-V_{2}(m)\leq-\frac{c}{b-\epsilon}$ and that it is optimal to change in state $2$ if and only if $V_{1}(m)-V_{2}(m)\geq\frac{c}{b-\epsilon}$. Thus, we know that choosing to change in both states is never optimal. Therefore we focus on the three potentially optimal strategies $\{\text{change}\}\times\{\text{stay}\}$, $\{\text{stay}\}\times\{\text{stay}\}$ and $\{\text{stay}\}\times\{\text{change}\}$. The expected discounted reward given these strategies is

[TABLE]

In order to understand which strategy is optimal, we compute the differences $V^{d}_{1}(m)-V^{d}_{2}(m)$:

[TABLE]

Using that $f_{\delta}(\cdot)$ is increasing and that $m_{1}=1-m_{2}$ we obtain for small enough $\delta>0$ that

[TABLE]

and analogously we obtain

[TABLE]

as well as

[TABLE]

As a next step we compute the fixed points given the three strategies $\{\text{change}\}\times\{\text{stay}\}$, $\{\text{stay}\}\times\{\text{stay}\}$ and $\{\text{stay}\}\times\{\text{change}\}$, noting that we never need to consider the fixed point given $\{\text{change}\}\times\{\text{change}\}$, as it is never optimal to randomize in both states. Since we face standard continuous time Markov chains, this task is simple and we obtain for each strategy a unique fixed point:

[TABLE]

Depending on the choice of parameters we have between one and five equilibria: For simplicity we write

[TABLE]

and note that we always have $d_{1}<d_{2}$.

We can now obtain the exact number and position of equilibria by coefficient comparison. The strategies that are used in mixed strategy equilibria are then obtained by solving the balance equation $m^{T}Q^{\pi}(m)=0$ with $m$ being the population distribution of the mixed strategy equilibrium. We will omit this last step, which is basically the task of solving a system of linear equations.

• (i)

If $d_{1}<\frac{\epsilon}{b+\epsilon}$ and $d_{2}<\frac{1}{2}$, then $x^{\{\text{stay}\}\times\{\text{change}\}}$ together with the deterministic strategy $\{\text{stay}\}\times\{\text{change}\}$ is the unique stationary mean field equilibrium.

• (ii)

If $d_{1}<\frac{\epsilon}{b+\epsilon}$ and $\frac{1}{2}\leq d_{2}\leq\frac{b}{b+\epsilon}$, then $x^{\{\text{stay}\}\times\{\text{stay}\}}$ together with the deterministic strategy $\{\text{stay}\}\times\{\text{stay}\}$ and $x^{\{\text{stay}\}\times\{\text{change}\}}$ together with the deterministic strategy $\{\text{stay}\}\times\{\text{change}\}$ are the deterministic equilibria. Furthermore, there is one mixed strategy equilibrium with $m_{1}=d_{2}$ that randomizes over $\{\text{stay}\}$ and $\{\text{change}\}$ in the second state. If $k_{2}\in\{\frac{1}{2},\frac{b}{b+\epsilon}\}$, then the mixed strategy equilibrium coincides with the pure strategy equilibrium with the same population distribution.

• (iii)

If $d_{1}<\frac{\epsilon}{b+\epsilon}$ and $\frac{b}{b+\epsilon}<d_{2}$, then the only equilibrium is given by $x^{\{\text{stay}\}\times\{\text{stay}\}}$ together with the deterministic strategy $\{\text{stay}\}\times\{\text{stay}\}$.

• (iv)

If $\frac{\epsilon}{b+\epsilon}\leq d_{1}\leq\frac{1}{2}$ and $d_{2}<\frac{1}{2}$, then $x^{\{\text{stay}\}\times\{\text{change}\}}$ together with the deterministic strategy $\{\text{stay}\}\times\{\text{change}\}$ and $x^{\{\text{change}\}\times\{\text{stay}\}}$ together with the deterministic strategy $\{\text{change}\}\times\{\text{stay}\}$ are the deterministic equilibria. Furthermore, there is a mixed strategy equilibrium at the point with $m_{1}=d_{1}$ that randomizes over $\{\text{stay}\}$ and $\{\text{change}\}$ in the first state. If $k_{1}=\frac{\epsilon}{b+\epsilon}$, then the mixed strategy equilibrium coincides with the pure strategy equilibrium with the same population distribution.

• (v)

If $\frac{\epsilon}{b+\epsilon}\leq d_{1}\leq\frac{1}{2}$ and $\frac{1}{2}\leq d_{2}\leq\frac{b}{b+\epsilon}$, then $x^{\{\text{stay}\}\times\{\text{change}\}}$ together with the deterministic strategy $\{\text{stay}\}\times\{\text{change}\}$, $x^{\{\text{stay}\}\times\{\text{stay}\}}$ together with the deterministic strategy $\{\text{stay}\}\times\{\text{stay}\}$ and $x^{\{\text{change}\}\times\{\text{stay}\}}$ together with the deterministic strategy $\{\text{change}\}\times\{\text{stay}\}$ are the deterministic equilibria. Furthermore, there is one mixed strategy equilibrium with $m_{1}=d_{1}$ that randomizes over $\{\text{stay}\}$ and $\{\text{change}\}$ in the first state and one mixed strategy equilibrium with $m_{1}=d_{2}$ that randomizes over $\{\text{stay}\}$ and $\{\text{change}\}$ in the second state. If $k_{1}\in\{\frac{\epsilon}{b+\epsilon},\frac{1}{2}\}$ or $k_{2}\in\{\frac{1}{2},\frac{b}{b+\epsilon}\}$, then the mixed strategy equilibrium coincides with the pure strategy equilibrium with the same population distribution.

• (vi)

If $\frac{\epsilon}{b+\epsilon}\leq d_{1}\leq\frac{1}{2}$ and $\frac{b}{b+\epsilon}<d_{2}$, then $x^{\{\text{stay}\}\times\{\text{stay}\}}$ together with the deterministic strategy $\{\text{stay}\}\times\{\text{stay}\}$ and $x^{\{\text{change}\}\times\{\text{stay}\}}$ together with the deterministic strategy $\{\text{change}\}\times\{\text{stay}\}$ are the deterministic equilibria. Furthermore, there is one mixed strategy equilibrium with $m_{1}=d_{1}$ that randomizes over $\{\text{stay}\}$ and $\{\text{change}\}$ in the first state. If $k_{1}\in\{\frac{\epsilon}{b+\epsilon},\frac{1}{2}\}$, then the mixed strategy equilibrium coincides with the pure strategy equilibrium with the same population distribution.

• (vii)

If $\frac{1}{2}<d_{1}$ and $\frac{1}{2}<d_{2}\leq\frac{b}{b+\epsilon}$, then $x^{\{\text{stay}\}\times\{\text{change}\}}$ together with the deterministic strategy $\{\text{stay}\}\times\{\text{change}\}$ and $x^{\{\text{change}\}\times\{\text{stay}\}}$ together with the deterministic strategy $\{\text{change}\}\times\{\text{stay}\}$ are the deterministic equilibria. Furthermore, there is a mixed strategy equilibrium at the point with $m_{1}=d_{2}$ that randomizes over $\{\text{stay}\}$ and $\{\text{change}\}$ in the second state. If $k_{2}=\frac{b}{b+\epsilon}$, then the mixed strategy equilibrium coincides with the pure strategy equilibrium with the same population distribution.

• (viii)

If $\frac{1}{2}<d_{1}$ and $\frac{b}{b+\epsilon}<d_{2}$, then then $x^{\{\text{change}\}\times\{\text{stay}\}}$ together with the deterministic strategy $\{\text{change}\}\times\{\text{stay}\}$ is the unique stationary mean field equilibrium.

6.2 A Simplified Corruption Model

We now consider a simplified version of the corruption model presented in Kolokoltsov and Malafeyev (2017): In their model a player can be in one of the three states honest ($H$), corrupt ($C$) and reserved ($R$). The corrupt players get a higher wage than the honest players, which in turn get a higher wage than the reserved players that have been convicted to be corrupt. For simplicity we set $w_{C}=10$, $w_{H}=5$ and $w_{R}=0$ and exclude the fine for being convicted, which is an additional feature in their model. The players can choose given that they are not reserved, whether they want to stay corrupt/honest or whether they want to switch behaviour. In this case they become honest/corrupt with the rate $b$. A player that is reserved is recovered with a fixed rate $r$ and we assume that he will then be honest. Additionally the model captures “social pressure” in two ways: First, the more players are corrupt the higher is the pressure (one cannot escape) to also become corrupt. Second, the more players are honest the higher is the rate to become convicted to be corrupt. In the model of Kolokoltsov and Malafeyev (2017) there is also a principal agent that convicts players, for simplicity we decided to ignore this feature of the model as well.

The formal characterization is given by $\mathcal{A}=\{\text{change},\text{stay}\}$ together with

[TABLE]

and $c_{\text{change}}=c_{\text{stay}}=(10,5,0)^{T}$, where all parameters $b$, $q_{\text{inf}}$, $q_{\text{soc}}$ and $r$ are strictly positive. A visualization of this model is given in Figure 1.

We start with computing the value function for given $m\in\mathcal{P}(\mathcal{S})$ as the unique solution of

[TABLE]

We directly see that one should choose to change in state $1$ if $V_{1}(m)\leq V_{2}(m)$ and to stay in state $1$ if $V_{1}(m)\geq V_{2}(m)$ and one should chose to change in state $2$ if $V_{1}(m)\geq V_{2}(m)$ and to stay in state $2$ if $V_{2}(m)\geq V_{1}(m)$. A straightforward calculation yields that the optimality sets are given by

[TABLE]

Note however, that depending on the choice of parameters the quantity $\frac{r+\beta}{q_{\text{soc}}}$ is greater than one, exactly one or less than one. In the first case only $\text{Opt}(\{\text{stay}\}\times\{\text{change}\})$ is non-empty, in the second case $\text{Opt}(\{\text{stay}\}\times\{\text{change}\})$ and $\text{Opt}(\{\text{stay},\text{change}\}\times\{\text{stay},\text{change}\})$ are non-empty and in the third case all optimality sets are non-empty.

As a second step we use the cut criterion from Theorem 5.1 for the set $\mathcal{T}=\{R\}$, which yields the equation $q_{\text{soc}}m_{H}m_{C}=rm_{R}$ for all stationary strategies. Together with the equation $1=m_{R}+m_{H}+m_{C}$ we obtain

[TABLE]

or equivalently $m_{C}=r(1-m_{H})/(q_{\text{soc}}m_{H}+r),$ note that this representations directly imply that $m_{C}\in[0,1]\Leftrightarrow m_{H}\in[0,1]$. Furthermore we obtain that the sum of $m_{C}$ and $m_{H}$ is always less than one, thus any mean field equilibrium is uniquely characterized by describing $m_{C}$. More precisely, any stationary mean field equilibrium has a distribution of the form

[TABLE]

It now remains to consider the fixed point problems given the possible optimal strategies: We start by noting that stationary points of the dynamics given $\{\text{stay}\}\times\{\text{change}\}$ exists whenever (5) and

[TABLE]

is satisfied, which is true if $m_{H}=0$ or $m_{C}=b/(q_{\text{soc}}-q_{\text{inf}})$. Thus whenever these points lie in $\text{Opt}(\{\text{stay}\}\times\{\text{change}\})$ or $\text{Opt}(\{\text{stay},\text{change}\}\times\{\text{stay},\text{change}\})$ we have a deterministic mean field equilibrium given the strategy $\{\text{stay}\}\times\{\text{change}\}$.

Similarly, stationary points of the dynamics given $\{\text{change}\}\times\{\text{stay}\}$ have to satisfy (5) and

[TABLE]

which is true if either $m_{C}=0$ or $m_{H}=b/(q_{\text{inf}}-q_{\text{soc}})$. Thus whenever these points lie in $\text{Opt}(\{\text{change}\}\times\{\text{stay}\})$ or $\text{Opt}(\{\text{stay},\text{change}\}\times\{\text{stay},\text{change}\})$ we have a deterministic mean field equilibrium given the strategy $\{\text{change}\}\times\{\text{stay}\}$.

When searching for stationary points given the dynamics of mixed strategies, which might be equilibria,we can restrict to those that lie inside the set $\text{Opt}(\{\text{stay},\text{change}\}\times\{\text{stay},\text{change}\})$. In this set all equilibria have to satisfy $m_{H}=\frac{r+\beta}{q_{\text{soc}}}$, that is

[TABLE]

It remains to check whether there is a strategy such that the point

[TABLE]

is indeed a fixed point for the individual dynamics equation given strategy $\pi$, which means that we have to find constants $\pi_{1,\text{change}}$ and $\pi_{2,\text{change}}$ that satisfy for this point

[TABLE]

which is parameter-dependent.

As in the previous example, we would now need to perform a case analysis to obtain the exact set of mean field equilibria for all possible equilibrium constellations. Additionally, we would need to solve the balance equations $m^{T}Q(m)=0$ for $\pi$ for any $m$ that is a candidates for randomized equilibria. Both tasks are simple, but tedious and we omit them here.

Appendix A Appendix

Proof of Lemma 5.3.

Since we face a conservative generator, we see by definition that $Q_{ij}\geq 0$ for all $i\neq j$ and that $\sum_{j\in\mathcal{S}}Q_{ij}=0$. Thus, all off-diagonal entries of $Q$ are non-negative and the row sum is always zero. Furthermore, by requiring irreducibility we do not have a row of zeros, thus the diagonal entries are strictly negative.

The matrix $Q_{SS}^{\prime}$ again has non-negative off-diagonal entries, strictly negative diagonal entries and row sum is less or equal zero. Furthermore the irreducibility of $Q$ implies that $Q_{iS}$ is non-zero for at least one $i\in\{1,\ldots,S-1\}$ and thus the row sum for at least one $i$ is strictly negative. We now show that there exists a vector $x\in\mathbb{R}^{S-1}$ such that $x\geq 0$ and $x_{i}>0$ for at least one $i\in\{1,\ldots,S-1\}$ such that $-Q_{SS}^{\prime}x>0$ (where the inequality signs hold pointwise).

For this we first note that for all $k\in\{1,\ldots,S-1\}$

[TABLE]

From which one directly sees that $(Q_{SS}^{\prime}x)_{k}$ is increasing in $x_{k}$ and decreasing in $x_{l}$ since $Q_{kl}>0$ for all $l\neq k$.

Define for all $x\in([0,\infty))^{S-1}$ the set $\mathcal{T}(x)=\{k\in\{1,\ldots,S-1\}:(Q_{SS}^{\prime}x)_{k}>0\}$ as the set of all indices where the $i$-th component of $Q_{SS}^{\prime}x$ is greater than zero. We will now define inductively an sequence $x^{n}\in\mathbb{R}^{S}$ a sequence of vectors such that for some $m\in\mathbb{N}$ we have $\mathcal{T}(x^{m})=\{1,\ldots,S-1\}$. We will start with the vector $x^{0}$ with a one at every component and construct $x^{n}$ in such a way that $\mathcal{T}(x^{n-1})\subsetneq\mathcal{T}(x^{n})$ for all $n\in\mathbb{N}$ and moreover $x_{i}^{n}>0$ for all $i\in\{1,\ldots,S-1\}$ and all $n\in\mathbb{N}_{0}$.

Starting with $x^{0}$ being the vector of ones directly implies that $\mathcal{T}(x^{0})\neq\emptyset$, as we have previously seen that the row sum, which is $(Q_{SS}^{\prime}x^{0})_{i}$ is positive for some index $i\in\{1,\ldots,S-1\}$.

In the $n$-th step ($n\in\mathbb{N}$) we check whether $\mathcal{T}(x^{n-1})=\{1,\ldots,S-1\}$. If this is the case we have shown that a vector with the desired properties exists, else there is an index $j\in\{1,\ldots,S-1\}\setminus\mathcal{T}(x^{n-1})$. In this case let $i\in\mathcal{T}(x^{n-1})$. Since our CTMC is irreducible the underlying transition graph is strongly connected, which implies that there is a path from $j$ to $i$. Let $\tilde{k}$ be the first node on this path that lies in $\mathcal{T}(x^{n-1})$ and let $k$ be its predecessor. Note that by definition of the transition graph $Q_{k\tilde{k}}>0$. Now let $x^{n}$ be as follows $x_{i}^{n}=x_{i}^{n-1}$ for all $i\in\{1,\ldots,S-1\}\setminus\{\tilde{k}\}$ and $x_{\tilde{k}}^{n}$ such that

[TABLE]

This is possible as $x_{\tilde{k}}^{n-1}>0$ and $(Q_{SS}^{\prime}x^{n-1})_{\tilde{k}}>0$ and moreover $(Q_{SS}^{\prime}x)_{\tilde{k}}$ is continuous and increasing in $x_{\tilde{k}}$.

It is obvious that $x^{n}_{l}>0$ for all $l\in\{1,\ldots,S-1\}$. It remains to check whether $m\in\mathcal{T}(x^{n-1})$ implies $m\in\mathcal{T}(x^{n})$ and to show that $k\in\mathcal{T}(x^{n})$, as this proves that $\mathcal{T}(x^{n-1})\subsetneq\mathcal{T}(x^{n})$.

As $Q_{SS}^{\prime}(x^{n})_{\tilde{k}}>0$ by construction it is obvious that $\tilde{k}\in\mathcal{T}(x^{n})$. For $m\neq\tilde{k}$ we see again as $(Q_{SS}^{\prime}x)_{m}$ is decreasing in $x_{\tilde{k}}$ and $x_{m}^{n}=x_{m}^{n-1}$ that

[TABLE]

Thus if $m\in\mathcal{T}(x^{n-1})$, then $m\in\mathcal{T}(x^{n})$.

For $m=k$ we have a strict inequality as $(x_{k}^{n}-x_{\tilde{k}}^{n})Q_{k\tilde{k}}>(x_{k}^{n-1}-x_{\tilde{k}}^{n-1})Q_{k\tilde{k}}$ since $Q_{k\tilde{k}}>0$ and $x_{\tilde{k}}^{n}<x_{\tilde{k}}^{n-1}$. Thus $(Q_{SS}^{\prime}x^{n})_{k}>(Q_{SS}^{\prime}x^{n-1})_{k}=0$, which directly implies $k\in\mathcal{T}(x^{n})$, thus $\mathcal{T}(x^{n-1})\subsetneq\mathcal{T}(x^{n})$.

Thus, we indeed obtained a vector $x\in\mathbb{R}^{S-1}$ such that $x\geq 0$ and $x_{i}>0$ for at least one $i\in\{1,\ldots,S-1\}$ such that $-Q_{SS}^{\prime}x>0$. Now we can conclude that it is a non-singular $M$-matrix in the sense of Berman and Plemmons (1979, Chapter 6) and furthermore that all eigenvalues of our matrix have positive real part. Therefore, all eigenvalues of $Q_{SS}$ have negative real part.

As the matrix $Q_{SS}$ is real, all complex eigenvalues appear in pairs of complex conjugates. As the product of a complex number and its complex conjugate is non-negative, we obtain that the determinant which can be computed as the product of all eigenvalues is as claimed since only the number of real eigenvalues or more specifically the parity of this number matters for the sign of the determinant. As the parity of the number of real eigenvalues always equals the parity of $S-1$, we conclude that the sign pattern of the determinant is as claimed. ∎