Asymptotic value in frequency-dependent games with separable payoffs: a differential approach

TL;DR

This paper investigates the long-term value of frequency-dependent zero-sum games with separable payoffs using a differential approach, establishing the existence and equivalence of asymptotic and continuous-time game values.

Contribution

It introduces a differential game framework for frequency-dependent payoffs and proves the existence and equality of asymptotic and continuous-time game values.

Findings

Asymptotic value exists in both n-stage and discounted games.

The differential game has a well-defined value despite irregularities.

Asymptotic and continuous-time game values coincide.

Abstract

We study the asymptotic value of a frequency-dependent zero-sum game with separable payoff following a differential approach. The stage payoffs in such games depend on the current actions and on a linear function of the frequency of actions played so far. We associate to the repeated game, in a natural way, a differential game and although the latter presents an irregularity at the origin, we prove that it has a value. We conclude, using appropriate approximations, that the asymptotic value of the original game exists in both the n-stage and the {\lambda}-discounted games and that it coincides with the value of the continuous time game.

Equations

$$g_t:=g(z_{t-1},i_t,j_t)=a_{i_t{j}_t}+h(z_{t-1}),$$

$$h(z):=\{\begin{array}{l}⟨H,\frac{z}{|z|}⟩,z\ne 0\\ 0,z=0,\end{array}$$

$$(n+1){\text{V}}_{n+1}(z)$$

$${\mathbf{V}}_{\lambda }\left(z\right)$$

$${\Pi }_t=\{z\in \mathcal{Z}:|z|=t\}.$$

$${\Lambda }_p(a):=\frac{1}{a}+\frac{1}{a+1}...+\frac{1}{a+p-1}=\sum _{k=0}^{p-1}\frac{1}{a+k}.$$

$$n{\text{V}}_n(z)=⟨K_{n,t},z⟩+C_{n,t}$$

$${\text{V}}_1(z)=⟨H,\frac{z}{|z|}⟩+\underset{u\in \Delta (I)}{\max }\underset{v\in \Delta (J)}{\min }\left(\sum _{ij}{u}_i{v}_j{a}_{ij}\right).$$

$$(m+1){\text{V}}_{m+1}(z)$$

$$\displaystyle=\left\langle\frac{H}{t}+K_{m,t+1},z\right\rangle+\textbf{val}\bigg{(}A+K_{m,t+1}\bigg{)}+C_{m,t+1},$$

$$(m+1)V_{m+1}(z)=⟨K_{m+1,t},z⟩+C_{m+1,t},$$

$$\mathcal{Q}_{N}:=\bigg{\{}q:q=\frac{z}{N},z\in\mathcal{Z}\bigg{\}},\hskip 56.9055pt\mathcal{I}_{N}:=\bigg{\{}0,\frac{1}{N},...,1\bigg{\}}.$$

$${\Psi }_N(t,q):=(1-t){\text{V}}_n(z),$$

$$\{\begin{array}{l}{\Psi }_N(t,q)=\frac{h(q)}{N}+\underset{u\in \Delta (I)}{\max }\underset{v\in \Delta (J)}{\min }\left(\sum _{i,j}{u}_i{v}_j\left(\frac{a_{ij}}{N}+{\Psi }_N\left(t+\frac{1}{N},q+\frac{e_{ij}}{N}\right)\right)\right),t\in {\mathcal{I}}_N\setminus \{1\}\\ {\Psi }_N(1,q)=0.\end{array}$$

$$\displaystyle 0=h(q)+\max\limits_{u\in\Delta(I)}\min\limits_{v\in\Delta(J)}\left(\sum\limits_{i,j}u_{i}v_{j}\left(a_{ij}+N\bigg{(}\Psi_{N}\left(t+\frac{1}{N},q+\frac{e_{ij}}{N}\right)-\Psi_{N}(t,q)\bigg{)}\right)\right).$$

$$\{\begin{array}{l}\frac{\partial \Psi }{\partial t}(t,q)+h(q)+\underset{u\in \Delta (I)}{\max }\underset{v\in \Delta (J)}{\min }\sum _{i,j}{u}_i{v}_j\left(a_{ij}+\frac{\partial \Psi }{\partial q_{ij}}(t,q)\right)=0,(t,q)\in [0,1)\times {\mathbb{R}}_{+}^{I\times J}\setminus \{0\},\\ \Psi (1,q)=0,q\in {\mathbb{R}}_{+}^{I\times J}\setminus \{0\}.\end{array}$$

$$\{\begin{array}{l}\frac{dq}{dt}(s)=\tilde{u}(s)\otimes \tilde{v}(s),s\in (t,1),\\ q(t)=q.\end{array}$$

$$\displaystyle g(q,u,v)=h(q)+\big{\langle}u\otimes v,A\big{\rangle}$$

$$h(q):=\{\begin{array}{l}⟨H,\frac{q}{|q|}⟩,q\ne 0\\ 0,q=0.\end{array}$$

$$G(t,q,\tilde{u},\tilde{v})=\underset{t}{\overset{1}{\int }}g(q(s),\tilde{u}(s),\tilde{v}(s))ds.$$

$${\tilde{v}}_1(s)={\tilde{v}}_2(s)\forall s\in [t,\tilde{t}]\Rightarrow \alpha [{\tilde{v}}_1(s)]=\alpha [{\tilde{v}}_2(s)]\forall s\in [t,\tilde{t}].$$

$$W^{-}(t,q)$$

$$W^{+}(t,q)$$

$${\mathcal{H}}^{-}(\xi ,q)$$

$${\mathcal{H}}^{+}(\xi ,q)$$

$$W(t,q)=\underset{\alpha \in {\mathcal{A}}_t}{\sup }\underset{\tilde{v}\in {\mathcal{V}}_t}{\inf }\left\{⟨H,\underset{t}{\overset{t+\delta }{\int }}\frac{q(s)}{|q|+s}ds⟩+⟨\underset{t}{\overset{t+\delta }{\int }}\alpha [\tilde{v}(s)]\otimes \tilde{v}(s)ds,A⟩+W^{\ast }\right\},$$

$$\displaystyle\begin{cases}\frac{\partial{W}}{\partial{t}}(t,q)+\mathcal{H}\big{(}{\nabla_{q}}W(t,q),q\big{)}=0,\hskip 17.07182pt(t,q)\in[0,1)\times\mathcal{Q}^{*},\\ W(1,q)=0,\hskip 5.69046ptq\in\mathcal{Q}^{*}.\end{cases}$$

$$|G(0,q,\tilde{u},\tilde{v})-G(0,0,\tilde{u},\tilde{v}|<\epsilon .$$

$$|G(q)-G(0)|$$

$$\displaystyle=2\left\|H\right\|_{\infty}\bigg{|}\bigg{(}\ln\big{(}1+|q|\big{)}-\ln\big{(}|q|\big{)}\bigg{)}\bigg{|}\left|q\right|$$

$$\displaystyle=2\left\|H\right\|_{\infty}\bigg{|}\ln\bigg{(}\frac{1+|q|}{|q|}\bigg{)}\bigg{|}\left|q\right|.$$

$$\displaystyle|G(q)-G(0)|\leq 4\left\|H\right\|_{\infty}\big{|}\ln(|q|)\big{|}|q|.$$

Full text

Keywords: stochastic game, frequency-dependent payoffs, continuous-time game, discretization, Hamilton-Jacobi-Bellman-Isaacs equation.

JEL Classification: C73 AMS Classification: 91A15 91A23 91A25

Introduction

Frequency-dependent games are repeated games where the stage payoffs depend on current actions and on frequency of past actions. The interpretation is that in such games, the actions undertaken by the players at each stage may generate externalities, which accumulate as the game unfolds. Stage payoffs may be frequency-dependent over time for several reasons. For instance, payoffs may change due to learning, habit formation, addiction, or satiation. The class of frequency-dependent games covers a wide variety of applications such as littering and pollution problems, the impact of human activities on other species and more generally on the environment. Frequency-dependent games were introduced by Brenner and Witt (2003) as a class of dynamic games with stage payoffs depending on the frequency of past actions. To the best of our knowledge, in the context of repeated games, Smale (1980) was the first to study dynamics that take the past into account. The former games consist in the repetition of a one-shot game in which the stage payoff depends on the choices of the players at the current stage, as well as on the relative frequencies of actions played at previous stages. An extensive review of this class of games and its applications can be found in Joosten et al. (2003), where the authors focus mostly on the infinite horizon general frequency-dependent games and in particular derive several Folk-Theorem-like. Moreover, Contou-Carrère (2011) recently studied some aspects of frequency-dependent games. The main insight of this work lies in the fact that on the one hand no uniform value exists even for a one-player game and on the other hand the asymptotic value exists although its convergence is not uniform in the state variable. More precisely, the author considers a particular case of the littering game (Joosten (2004)), in which the decision maker has two actions, one that deteriorates the environment, and the other one that preserves it; the littering action is a dominant action in each stage game, but the repeated use of this action produces a lasting effect on the environment so that all future payoffs are decreased. The author proves in a game with a given length $n\in\mathbb{N}^{*}$ that the unique optimal strategy consists in using the non littering action from stage $t=1$ up to some stage $t^{*}(n)$ and then starting to use the littering action until the end of the play. Since the time of switching from one action to the other depends on the length of the game, one can prove that the uniform value does not exist in the frequency-dependent control problem. Nevertheless, the fraction of time $t^{*}(n)/n$ converges when $n$ goes to infinity so that in particular the asymptotic value $\lim\textbf{V}_{n}(z)$ exists and is independent of the initial state $z$, although this convergence is not uniform in the state.

In order to characterize the Nash equilibrium payoffs of a long game, the study of the zero-sum case seems to be necessary. In this paper, we investigate the value of a class of two-player zero-sum frequency-dependent games with finite action sets $I$ and $J$ respectively, namely the subclass of frequency-dependent games with separable payoffs333As far as we know, in general stochastic games, payoffs with separable structure first appeared in Parthasarathy et al. (1984). The authors study games with transitions independent of the current state.. This means that the stage payoff is the sum of two parts, one part is derived from the current actions and the other one depends linearly on the frequency of the past actions. This game can be viewed as a stochastic game with countable state space, namely $\mathbb{N}^{I\times J}$ and deterministic transitions. The current state at the $n$-th stage is the aggregate past matrix, i.e., it reflects how many times each action profile has been selected in the previous $n-1$ stages. Player 1 maximizes and Player 2 minimizes the average payoff in the first $n$ stages and the game is played under perfect-monitoring, meaning that both players know the current state as well as the entire history, i.e., the state visited and the action pair played at each of the preceding stages. Since it is already known from the study of the one-player game that no uniform value exists, our main focus will be the existence of the asymptotic value of this game. We treat in parallel both the average and the discounted case. Note that the convergence being non-uniform in the state variable (see Contou-Carrère (2011)), we cannot rely on the Tauberian theorem of Ziliotto (2016) to deduce the existence of one of the limits from the existence of the other.

The traditional approach to the sequence $(\textbf{V}_{n})_{n\in\mathbb{N}^{*}}$ is through the study of the so-called recursive equation (see Mertens et al. (2015), Theorem 3.2, p.158). However, as in the case of many repeated games, it seems difficult to derive the asymptotic behavior directly from this formula. Therefore, we switch to a differential approach in the sense that we associate to the repeated game, a differential game played over $[0,1]\times\mathbb{R}^{I\times J}$. Indeed, by some heuristic reasoning it is possible to conjecture as a limit of the recursive equation, a hypothetical partial differential equation (PDE) that governs the evolution of the value. It turns out that this is precisely the Hamilton-Jacobi-Bellman-Isaacs (HJBI) equation of some differential game and furthermore that the value of this continuous game is closely related to the value of our repeated game. However, an important difficulty arises due to an irregularity of the payoff function at the origin and it is precisely at the origin where our analysis has to be done. Everywhere but at the origin, regularity conditions are satisfied by the payoff and dynamics functions and since the Isaacs condition holds true, i.e., lower and upper Hamiltonians coincide, by Evans and Souganidis (1984)444Existence of the value follows from the standard comparison and uniqueness theorems for viscosity solutions presented in Crandall and Lions (1983). and Souganidis (1999), it follows existence of the value in the differential game. Moreover, the value is characterized as the unique viscosity solution in the space of bounded, continuous functions of the HJBI equation with a boundary condition. Despite the irregularity at the origin, we prove existence of the value in the differential game starting at $(0,0)$. In order to compare the values of the repeated game with that of the differential game, we proceed by discretization. The previously mentioned irregularity at the origin makes it impossible to apply the usual methods of approximations. We adapt the methods of Souganidis (1999) for finite horizon differential games (see Bardi and Capuzzo-Dolcetta (2008) for infinite horizon555The authors prove that under some regularity conditions on the payoff and dynamics functions, the discrete values converge to the values of the continuous time game as the mesh of the discretization tends to [math]. These approximations do not converge in general if the value function is discontinuous.) so that they can fit our context. We prove that $\textbf{V}_{n}$ as $n$ tends to infinity, and $\textbf{V}_{\lambda}$ as $\lambda$ goes to zero, both converge to the same limit which is precisely the value of the differential game starting at the origin.

In the literature, the use of differential games to study the asymptotic value of a repeated game is not new. A differential approach first appeared in Vieille (1992) to study weak approachability. An approach similar to ours, has been proposed by Laraki (2002) to prove existence of the asymptotic value in $n$-stage and $\lambda$-discounted repeated games with incomplete information on one side. Cardaliaguet et al. (2012) achieve a transposition to discrete time games of the numerical schemes used to approximate the value function of differential games via viscosity solution arguments, presented in Barles and Souganidis (1991). The authors prove existence of the asymptotic value in absorbing, splitting and incomplete information games, where convergence is uniform in the state variable. Our approach differs from all these literatures by the nature of the state space of the continuous game and chiefly in that, due to the irregularity at the origin in our setting, existence of the value in the continuous game is not straightforward. Since in our model, the state space is countable following their approach would lead us to an infinite dimensional state space in the associated differential game. As a consequence, the way we associate the differential game to the repeated game is quite different from theirs.

Structure of the paper. The remainder of the paper is organized as follows. In Section 1, we give the description of a two-player frequency-dependent zero-sum game and we provide properties of the $n$-stage value function, which will be useful, in the sequel. In Section 2, starting from the recursive formula satisfied by the value, we heuristically derive a PDE. Then, we define the associated differential game and prove existence of the value in the differential game played over $[0,1]$ and starting at initial state [math]. We then provide its uniformly and $\lambda$-discounted discretized versions. In Section 3, we conclude by identifying the value of the continuous time game, as the limit value of the $n$-stage and the $\lambda$-discounted frequency-dependent games. In Section 4 we conclude.

1 The frequency-dependent game and preliminary results

In this section, we describe our model and we give some preliminary results.

1.1 Definitions

Let $I,J$ be finite sets and denote the space of real matrices with $|I|$ rows and $|J|$ columns by $\mathcal{M}^{I\times J}$. The notation $(e_{ij})_{ij}$ stands for the canonical basis in $\mathbb{R}^{I\times J}$, i.e., $e_{ij}=(\delta_{i}^{i^{\prime}}\delta_{j}^{j^{\prime}})_{i^{\prime}j^{\prime}}$, where $\delta_{k}^{k^{\prime}},k\in\{i,j\}$ denotes the Kronecker delta. Let $A=[a_{ij}]$ and $H$ be two elements of $\mathcal{M}^{I\times J}$ and let $z_{0}\in\mathcal{Z}:=\mathbb{N}^{I\times J}$. A frequency-dependent zero-sum repeated game with initial state $z_{0}$ is a dynamic game played by steps as follows:

At stage $t=1,2,...$, Player 1 and 2 simultaneously and independently choose an action in their own set of actions, $i_{t}\in I$ and $j_{t}\in J$ respectively. The stage payoff of Player 1 is given by:

[TABLE]

where $z_{t}=z_{0}+e_{i_{1}j_{1}}+...+e_{i_{t}j_{t}}$ and for any $z\in\mathcal{Z}$,

[TABLE]

where $\left\langle\cdot,\cdot\right\rangle$ denotes the canonical inner product in $\mathbb{R}^{I\times J}$ and $|\cdot|$ stands for $\left\|\cdot\right\|_{1}$. The payoff of Player 2 is the opposite of that of Player 1. We assume perfect monitoring of past actions by both players.

Notation. In the sequel, we use the notations: $\mathbb{N}^{*}=\mathbb{N}\setminus\{0\}$ and $\mathbb{R}_{+}^{*}=\mathbb{R}_{+}\setminus\{0\}$.

1.2 The values of $\Gamma_{N}(z_{0})$ and $\Gamma_{\lambda}(z_{0})$

Given $z_{0}\in\mathcal{Z}$, for any $N\in\mathbb{N}^{*}$ and any $\lambda\in(0,1)$, we will be interested in the finite $N$-stage and $\lambda$-discounted games of initial state $z_{0}$, denoted by $\Gamma_{N}(z_{0})$ and $\Gamma_{\lambda}(z_{0})$ respectively. A play is given by $\omega:=(i_{t},j_{t})_{t\in\mathbb{N}^{*}}$ and the induced rewards in the game of initial state $z_{0}$, is $\gamma_{N}(z_{0},\omega)=(1/N)\sum_{t=1}^{N}g_{t}$, (resp. $\gamma_{\lambda}(z_{0},\omega)=\sum_{t=1}^{\infty}\lambda(1-\lambda)^{t-1}g_{t}$). Note that, due to the nature of the transition in the state space, announcing the selected moves publicly also reveals the state variable to the players. Therefore, we will denote by $\mathbf{H}_{t}=\mathcal{Z}\times(I\times J)^{t-1}$ the set of histories at stage $t$ and $\mathbf{H}=\cup_{t\geq 0}\mathbf{H}_{t}$ will denote the set of all histories. $\Delta(I)$ and $\Delta(J)$ are the sets of mixed moves of Player 1 and Player 2 respectively. A behavioral strategy for Player 1 is a family of maps $\sigma=(\sigma_{t})_{t\geq 1}$, such that $\sigma_{t}:\mathbf{H}_{t}\rightarrow\Delta(I)$. Similarly, a behavioral strategy for Player 2 is a family of maps $\tau=(\tau_{t})_{t\geq 1}$, where $\tau_{t}:\mathbf{H}_{t}\rightarrow\Delta(J)$. $\Sigma$ and $T$ denote the sets of behavioral strategies of Player 1 and Player 2, respectively. Given $z_{0}\in\mathcal{Z}$, each strategy profile $(\sigma,\tau)$ induces a unique probability distribution $\mathbb{P}^{z_{0}}_{\sigma,\tau}$ on the set $\mathcal{Z}\times(I\times J)^{\infty}$ of plays (endowed with the $\sigma$-field generated by the cylinders). $\mathbb{E}^{z_{0}}_{\sigma,\tau}$ stands for the corresponding expectation. We study the games $\Gamma_{N}(z_{0})$ and $\Gamma_{\lambda}(z_{0})$ in which the payoff of Player 1 is given by \gamma_{N}(z_{0},\sigma,\tau)=\mathbb{E}^{z_{0}}_{\sigma,\tau}\big{(}\frac{1}{N}\sum_{t=1}^{N}g_{t}\big{)} and $\gamma_{\lambda}(z_{0},\sigma,\tau)=\mathbb{E}^{z_{0}}_{\sigma,\tau}\left(\sum_{t=1}^{\infty}\lambda(1-\lambda)^{t-1}g_{t}\right)$ respectively. Existence of the value in $\Gamma_{N}(z)$ and $\Gamma_{\lambda}(z)$ in mixed strategies follows from the minmax theorem of von Neumann (1928) and since the game is played under perfect-recall by Kuhn’s theorem the value can be achieved by using behavioral strategies. The $N$-stage and $\lambda$-discounted values are given by $\textbf{V}_{N}(z_{0})=\sup_{\sigma\in{\Sigma}}\inf_{\tau\in T}\gamma_{N}(z_{0},\sigma,\tau)$ and $\textbf{V}_{\lambda}(z_{0})=\sup_{\sigma\in{\Sigma}}\inf_{\tau\in T}\gamma_{\lambda}(z_{0},\sigma,\tau)$ respectively. By Mertens et al. (2015) (Theorem 3.2, p.158) given $(n,\lambda)\in\mathbb{N}^{*}\times(0,1]$ and a state $z\in\mathcal{Z}$, $\textbf{V}_{n}$ and $\textbf{V}_{\lambda}$ satisfy the following recursive formulas:

[TABLE]

[TABLE]

In the remainder of this section, given $n\in\mathbb{N}^{*}$, we provide a formula for the value of the $n$-stage game. This formula is too complex to allow the study of the limit, nevertheless it sheds a light on the asymptotic behavior of the value.

Notation. We use the following notations:

• •

Given $t\in\mathbb{N}$, let $\Pi_{t}$ denote the subset of the state space $\mathcal{Z}$ defined as follows:

[TABLE]

• •

We denote the $\max\min$ operator by val.

• •

For any $(a,p)\in\mathbb{R}^{*}_{+}\times\mathbb{N}^{*}$, we put :

[TABLE]

Proposition 1.1**.**

For all $n\in\mathbb{N}^{*}$, for all $t\in\mathbb{N}^{*}$, let $K_{n,t}\in\mathcal{M}^{I\times J}$ and $C_{n,t}\in\mathbb{R}$, such that $K_{n,t}=\Lambda_{n}(t)H$ and C_{n,t}=\sum_{k=1}^{n-1}{\bf val}\big{(}A+K_{n-k,t+k}\big{)}. Then, for all $z\in\Pi_{t}$

[TABLE]

Proof.

We proceed by induction on the variable $n$: For $n=1$, for any $t\in\mathbb{N}^{*}$ and any $z\in\Pi_{t}$:

[TABLE]

Then, $\textbf{V}_{1}(z)=\left\langle K_{1,t},z\right\rangle+C_{1,t}$, where $K_{1,t}=\frac{H}{t}$ and $C_{1,t}=\textbf{val}(A)$.

The recursive formula (1.1) and the induction hypothesis for $n=m$ implies, for all $z\in\Pi_{t}$:

[TABLE]

the middle equality folows from the inner product properties and the fact that $\sum_{ij}u_{i}v_{j}=1$, and the last one from the val operator properties. Hence,

[TABLE]

where $K_{m+1,t}=\frac{H}{t}+K_{m,t+1}$ and C_{m+1,t}=\textbf{val}\big{(}A+K_{m,t+1}\big{)}+C_{m,t+1}, This concludes the proof of the assumption. The rest is routine algebra. Note that $\Lambda_{m+1}(t)=\Lambda_{m}(t+1)+\frac{1}{t}$. ∎

Corollary 1.2**.**

Let $\rho\in(0,1)$. If $N\rightarrow+\infty$ and $\frac{n}{N}\rightarrow\rho$, then $\lim K_{n,N-n}=-H\ln(1-\rho)$.

Proof.

Writing $\Lambda_{n}(N-n)=\sum_{k=1}^{N-1}\frac{1}{k}-\sum_{k=1}^{N-n-1}\frac{1}{k}$, the limit follows readily from the fact that the sequence $\sum_{k=1}^{n}\frac{1}{k}-\ln n$ converges to the Euler constant $\gamma$ when $n$ goes to infinity. ∎

2 A differential approach

Given the current stage of the game of total length $N$, the number of stages until the end of the game is denoted by $n$ and in view of Corollary 1.2, the relevant asymptotic filter for the convergence is not that of $N\rightarrow+\infty$ but the one of $\frac{n}{N}\rightarrow\rho\in(0,1)$, where $\rho$ is the fraction of the remaining game. Hence, it seems natural to introduce a continuous version of the dynamic game. To begin with, moving from the recursive formula obtained in (1.1), we heuristically derive a PDE (Section 2.1). It turns out that the latter is precisely the HJBI equation of some differential game that we shall define.

For any $N\in\mathbb{N}^{*}$, we define the quotient state space and the uniform partition of $[0,1]$:

[TABLE]

2.1 The heuristic PDE and the differential game $\mathcal{G}(t,q)$

We define the function $\Psi_{N}:\mathcal{I}_{N}\times\mathcal{Q}_{N}\rightarrow\mathbb{R}$, such that

[TABLE]

where $z=Nq$ and $n=N(1-t)$. Then, $\Psi_{N}$ satisfies for any $q\in\mathcal{Q}$,

[TABLE]

The first formula of (2.2), for any $t\in\mathcal{I}_{N}\setminus\{1\}$ can be written equivalently:

[TABLE]

When $N\rightarrow+\infty$, we heuristically assume that there exists a sufficiently differentiable function $\Psi:[0,1]\times\mathbb{R}^{I\times J}_{+}\setminus\{0\}\rightarrow\mathbb{R}$ (the limit of $\Psi_{N}$), which will therefore satisfy the following (PDE) with boundary condition of (2.2):

[TABLE]

The differential game. Given $(t,q)\in[0,1]\times\mathbb{R}^{I\times J}_{+}$, we define a differential zero-sum game, denoted by $\mathcal{G}(t,q)$ starting at time $t$ with initial state $q$. It consists of:

• •

The state space $\mathcal{Q}=\mathbb{R}^{I\times J}_{+}$.

• •

The time interval of the game $T=[t,1]$.

• •

Player 1 uses a measurable control $\tilde{u}:[t,1]\rightarrow\Delta(I)$ and his control space is $\mathcal{U}_{t}$. Player 2 uses a measurable control $\tilde{v}:[t,1]\rightarrow\Delta(J)$ and his control space is $\mathcal{V}_{t}$. For $t=0$, we use the notation $\mathcal{U}:=\mathcal{U}_{0}$ and $\mathcal{V}:=\mathcal{V}_{0}$.

• •

If Player 1 uses $\tilde{u}$ and Player 2 uses $\tilde{v}$, then the dynamics in the state space is defined as follows:

[TABLE]

Clearly, the dynamics is driven by a bounded, continuous function, which is Lipschitz in $q$ and thus (2.5) admits a unique solution.

• •

The running payoff at time $s\in[t,1]$ that Player 1 receives from Player 2 is given by $g:\mathcal{Q}\times\Delta(I)\times\Delta(J)\rightarrow\mathbb{R}$ and defined as:

[TABLE]

where,

[TABLE]

It is easy to see that $g$ is bounded by $\left\|H\right\|_{\infty}+\left\|A\right\|_{\infty}$ and since $q:[t,1]\rightarrow\mathcal{Q}$ is a differentiable function of time (see (2.5)), $g$ is differentiable on $\mathcal{Q}\setminus\{0\}$.

• •

The payoff associated to the pair of controls $(\tilde{u},\tilde{v})\in\mathcal{U}_{t}\times\mathcal{V}_{t}$ that Player 2 pays to Player 1 at time $1$ is given by:

[TABLE]

Following Varaiya (1967), Roxin (1969) and Elliott and Kalton (1972), we allow the players to update their controls using non-anticipative strategies. A non-anticipative strategy for Player 1 is a map $\alpha:\mathcal{V}_{t}\rightarrow\mathcal{U}_{t}$ such that for any time $\tilde{t}>t$,

[TABLE]

The definition of non-anticipative strategies for Player 2 is analogous. Denote by $\mathcal{A}_{t}$ and $\mathcal{B}_{t}$ the sets of non-anticipative strategies of the players respectively and let us put $\mathcal{A}:=\mathcal{A}_{0}$ and $\mathcal{B}:=\mathcal{B}_{0}$. With respect to this notion of strategies, the lower and upper values are defined as follows:

[TABLE]

When both functions coincide, we say that the game $\mathcal{G}(t,q)$ has a value, denoted by $W(t,q)$.

Following Cardaliaguet (2000) and Bardi and Capuzzo-Dolcetta (2008), the lower and upper hamiltonian functions of the game $\mathcal{G}(t,q)$, $\mathcal{H}^{\pm}:\mathcal{Q}\times\mathcal{Q}\rightarrow\mathbb{R}$ are given by:

[TABLE]

Notation. In the sequel, $\mathcal{Q}^{*}$ stands for $\mathcal{Q}\setminus\{0\}$.

Given $(t,q)\in[0,1]\times\mathcal{Q}^{*}$, we have: (i) $\Delta(I)$ and $\Delta(J)$ are compact sets; (ii) the dynamics in the state space (2.5) and the running payoff (2.6) are bounded, continuous in all their variables and Lipschitz in the state variable $q$ functions; (iii) from the minmax theorem in von Neumann (1928), it clearly follows that the Isaacs condition, i.e., $\mathcal{H}^{-}=\mathcal{H}^{+}$ holds true (see (2.1)). Then, by Evans and Souganidis (1984) and Souganidis (1999), the differential game $\mathcal{G}(t,q)$ starting at time $t\in[0,1)$ with initial state $q\in\mathcal{Q}^{*}$ admits a value, denoted by $W(t,q)$. Moreover, the authors characterize666In the literature, the lower and upper values of a differential game have been first characterized by means of DPP in Elliott and Kalton (1974). the value by means of the Dynamic Programming Principle (DPP). Namely, for all $(t,q)\in[0,1)\times\mathcal{Q}^{*}$ and all $\delta\in(0,1-t]$, we have:

[TABLE]

where W^{*}:=W\big{(}t+\delta,q(t+\delta)\big{)} with $q(t+\delta)=q+\int\limits_{t}^{t+\delta}\alpha[\tilde{v}(s)]\otimes\tilde{v}(s)ds$.

Furthermore under the preceding assumptions, $W(t,q)$ is the unique solution in the space of real-valued, bounded, continuous functions defined over $[0,1]\times\mathcal{Q}^{*}$ of the following HJBI equation:

[TABLE]

where, $\mathcal{H}:=\mathcal{H}^{-}=\mathcal{H}^{+}$ is the hamiltonian defined earlier. Consequently, one can identify the PDE obtained in (2.4) with the HJBI equations of (2.10).

2.2 Existence of the value in $\mathcal{G}(0,0)$

In this section, we extend the results of the differential game $\mathcal{G}(t,q)$ over the set $[0,1]\times\mathcal{Q}$. Precisely, we show that $W(0,q)$ admits a limit as $q$ tends to [math] and we further establish that such limit is the value of the game starting at $(0,0)$, which therefore exists. The idea of the proof lies in the consideration of $\frac{\varepsilon}{2}$-optimal strategies in $\mathcal{G}(0,q)$ that are $\varepsilon$-optimal in $\mathcal{G}(0,0)$.

Lemma 2.1**.**

Let $q\in\mathcal{Q}^{*}$ and $(\tilde{u},\tilde{v})\in\mathcal{U}\times\mathcal{V}$. Denote by $q(\cdot)$ and $\tilde{q}(\cdot)$ the trajectories with initial conditions $q(0)=q$ and $\tilde{q}(0)=0$ obtained from (2.5). Then, for any $s\in[0,1]$, we have: $\left|h\left(q(s)\right)-h\left(\tilde{q}(s)\right)\right|\leq 2\left\|H\right\|_{\infty}\hskip 1.42271pt\frac{\left|q\right|}{\left|q(s)\right|}$.

Proof.

If $\tilde{q}(s)=0$, we get $h(\tilde{q}(s))=0$ and thus, $|h(q(s)|\leq\|H\|_{\infty}$. For any $s\in[0,1]$, such that $\tilde{q}(s)\neq 0$, since $q(s)\neq 0$, it is elementary that $\left|\frac{q(s)}{\left|q(s)\right|}-\frac{\tilde{q}(s)}{\left|\tilde{q}(s)\right|}\right|\leq 2\frac{|q(s)-\tilde{q}(s)|}{|q(s)|}$. Since the controls of the players depend only on the time variable, by the ordinary differential equation (2.5), for all $s\in[0,1]$, $\left|q(s)-\tilde{q}(s)\right|=\left|q\right|$. Then, we obtain \big{|}\frac{q(s)}{\left|q(s)\right|}-\frac{\tilde{q}(s)}{\left|\tilde{q}(s)\right|}\big{|}\leq 2\hskip 1.42271pt\frac{\left|q\right|}{\left|q(s)\right|}. Hence, |h(q(s))-h(\tilde{q}(s))|=\big{|}\big{\langle}H,\frac{q(s)}{|q(s)|}-\frac{\tilde{q}(s)}{|\tilde{q}(s)|}\big{\rangle}\big{|}\leq 2\left\|H\right\|_{\infty}\hskip 1.42271pt\frac{\left|q\right|}{\left|q(s)\right|}. ∎

Lemma 2.2**.**

For any $\varepsilon>0$, there exists $\eta\in(0,\frac{1}{4})$, such that for any $q\in\mathcal{Q}^{*}$ with $|q|=\eta$ and any $(\tilde{u},\tilde{v})\in\mathcal{U}\times\mathcal{V}$, we have:

[TABLE]

Proof.

Let us put $G(q):=G(0,q,\tilde{u},\tilde{v})$ and $G(0):=G(0,0,\tilde{u},\tilde{v})$ and we further denote by $q(\cdot)$ and $\tilde{q}(\cdot)$ the trajectories with initial conditions $q(0)=q$ and $\tilde{q}(0)=0$ obtained from (2.5). Then,

[TABLE]

For all $s\in[0,1]$, it holds true that $|q(s)|=|q|+s$. By Lemma 2.1, we get:

[TABLE]

If $|q|<\frac{1}{4}$, we claim that \big{|}\ln(1+|q|)\big{|}\leq|\ln(|q|)|. Indeed, since $|q|<\frac{1}{4}$, we have $|q|(1+|q|)<1$ and thus, $1<1+|q|<\frac{1}{|q|}$, so that $0\leq|\ln(1+|q|)|<|\ln(|q|)|$. Then, for any $q\in\mathcal{Q}^{*}$ such that $|q|<\frac{1}{4}$, we have:

[TABLE]

To conclude, for any $\varepsilon>0$, choose $\eta\in(0,\frac{1}{4})$, such that $\eta|\ln(\eta)|<\frac{\varepsilon}{4\left\|H\right\|_{\infty}+1}$ and the result is immediate. ∎

Theorem 2.3**.**

The game $\mathcal{G}(0,0)$ has a value, $W(0,0)=\lim\limits_{q\rightarrow 0}W(0,q)$.

Proof.

Let $\varepsilon>0$ and fix $\eta\in(0,\frac{1}{4})$, such that $\eta|\ln(\eta)|<\frac{\varepsilon}{4(4\left\|H\right\|_{\infty}+1)}$. Since the game $\mathcal{G}(0,q)$ admits a value for any $q\in\mathcal{Q}^{*}$, consider an $\frac{\varepsilon}{4}$-optimal non-anticipative strategy for Player 1 in $\mathcal{G}(0,q)$, where $|q|=\eta$, i.e., a measurable function $\alpha(\cdot)$, such that for all $s\in[0,1]$, $\alpha[\tilde{v}(s)]\in\mathcal{U}$. Then, for any $\tilde{v}\in\mathcal{V}$, we have:

[TABLE]

where the first inequality follows by Lemma 2.2 and the second inequality is due to $\alpha(\cdot)$ being an $\frac{\varepsilon}{4}$-optimal strategy. Reversing the roles of the players and following similar arguments we get that for any $\tilde{u}\in\mathcal{U}$, $G(0,0,\tilde{u},\beta[\tilde{u}])<G(0,q,\tilde{u},\beta[\tilde{u}])+\frac{\varepsilon}{4}<W^{+}(0,q)+\frac{\varepsilon}{4}+\frac{\varepsilon}{4}$, where $\beta(\cdot)$ is an $\frac{\varepsilon}{4}$-optimal non-anticipative strategy of Player 2 in $\mathcal{G}(0,q)$. Since the value exists in $\mathcal{G}(0,q)$ for any $q\in\mathcal{Q}^{*}$, we have:

[TABLE]

and we therefore get $|W^{-}(0,0)-W^{+}(0,0)|<\varepsilon$, which proves existence of the value in $\mathcal{G}(0,0)$ since the inequality holds true for any positive $\varepsilon$. ∎

2.3 The discretized game $\mathcal{G}_{\mathcal{P}}\left(t_{0},q_{0}\right)$

In this section, we introduce a discrete version of our differential game and prove a strong relation between its value and the value of the original repeated game. For that purpose, we next consider subdivisions of $[0,1]$ and we define a family of discretized games played on them.

• •

For all $t_{0}\in[0,1)$, $\mathcal{P}$ stands for any countable subdivision of $[t_{0},1]$ and if $\mathcal{P}$ is finite let $\omega_{\mathcal{P}}$ denote the number of intervals of such subdivision, otherwise we put $\omega_{\mathcal{P}}=\infty$.

• •

Given $N\in\mathbb{N}^{*}$, let $\mathcal{P}_{N}=(t_{k}^{N})_{0\leq k\leq N}$, where $t_{k}^{N}:=\frac{k}{N}$ stands for the uniform subdivision of $[0,1]$ in $N$ intervals. We will also use the notation \mathcal{P}_{N}=\big{(}t_{n}^{N}\big{)}_{0\leq n\leq N} for $n=N-k$.

• •

Given $\lambda\in(0,1)$, $\mathcal{P}_{\lambda}=(t_{k}^{\lambda})_{k\geq 0}$ stands for the countable subdivision of $[0,1]$ induced by the discount factor $\lambda$, such that $t_{0}^{\lambda}=0$, $t_{1}^{\lambda}=\lambda$, $t_{k}^{\lambda}:=\lambda+...+\lambda(1-\lambda)^{k-1}$, for $k\geq 1$ and $t_{\infty}^{\lambda}=1$.

• •

By $\pi_{k}:=t_{k+1}-t_{k}$ is denoted the $k$-th increment and $|\mathcal{P}|$ stands for the mesh of the subdivision $\mathcal{P}$, i.e., $|\mathcal{P}|=\sup\limits_{k}|\pi_{k}|$.

Given $\mathcal{P}$, for all $(t_{0},q_{0})\in[0,1)\times\mathcal{Q}$ we associate to $\mathcal{G}\left(t_{0},q_{0}\right)$ a discrete time game adapted to the subdivision $\mathcal{P}$ denoted by $\mathcal{G}_{\mathcal{P}}(t_{0},q_{0})$. Such a discrete time game starts at time $t_{0}$, has initial state $q_{0}\in\mathcal{Q}$ and is repeated $\omega_{\mathcal{P}}$ times. At time $t_{k}\in\mathcal{P}$, both players observe the current state $q_{k}$ and choose simultaneously and independently actions $u_{k+1}$ and $v_{k+1}$ in $\Delta(I)$ and $\Delta(J)$ respectively. The control sets are denoted by $\Delta(I)^{\omega_{\mathcal{P}}}$ and $\Delta(J)^{\omega_{\mathcal{P}}}$, indicating that players now choose piecewise constant functions defined over the $\omega_{\mathcal{P}}$-times Cartesian product of their corresponding mixed strategy sets. We will use the notation $\hat{u}=(u_{k})_{k=1}^{\omega_{\mathcal{P}}}$ and $\hat{v}=(v_{k})_{k=1}^{\omega_{\mathcal{P}}}$. The state evolves according to:

[TABLE]

At stage $k$, the expected payoff that Player 1 receives from Player 2 is given by:

[TABLE]

and given $(\hat{u},\hat{v})\in\Delta(I)^{h_{\mathcal{P}}}\times\Delta(J)^{h_{\mathcal{P}}}$, the total payoff of the game is

[TABLE]

Given $\mathcal{P}$, for all $(t_{0},q_{0})\in[0,1)\times\mathcal{Q}$, the game $\mathcal{G}_{\mathcal{P}}(t_{0},q_{0})$ admits a value. Following Friedman (1970), the value of the game denoted by $W_{\mathcal{P}}(t_{0},q_{0})$ is characterized by means of discrete version of the HJBI equations (2.10):

[TABLE]

We will refer to this equation as the dicrete Dynamic Programming Principle that will be abbreviated to discrete DPP.

In the sequel, we compare the $n$-stage (resp. the $\lambda$-discounted) game with an appropriate discretization of the differential game. Note that they differ essentially in the nature of their outcome space: While in the discretized game $\mathcal{G}_{\mathcal{P}}\left(t_{0},q_{0}\right)$, the play generated by pure strategies is deterministic and lives in $\mathbb{R}^{I\times J}$, in the original game $\Gamma_{N}(z_{0})$ (resp. $\Gamma_{\lambda}(z_{0})$) the play generated by behavioral strategies is random and takes its values in a discrete subset of $\mathbb{R}^{I\times J}$.

2.4 Coincidence of $\Psi_{N}$ and $W_{\mathcal{P}_{N}}$

We first prove that $\Psi_{N}$ preserves a very similar property to the one satisfied by $\textbf{V}_{n}$ in Proposition 1.1 and we then show $\Psi_{N}=W_{\mathcal{P}_{N}}$. Recall that the map $\Psi_{N}:\mathcal{P}_{N}\times\mathcal{Q}_{N}\rightarrow\mathbb{R}$ has been defined by (2.1) and is characterized by the recursive formula (2.2). It is then clear that an extension of $\Psi_{N}$ to a map $\Psi_{N}:\mathcal{P}_{N}\times\mathcal{Q}\rightarrow\mathbb{R}$ is obtained if we define it by the same recursive formula and terminal condition; namely for any $q\in\mathcal{Q}$,

[TABLE]

Proposition 2.4**.**

Let $N\in\mathbb{N}^{*}$. There exists a sequence $(k_{n,s},c_{n,s})\in\mathcal{M}^{I\times J}\times\mathbb{R}$ where $n\in\{0,...,N\},s\in\mathbb{R}_{+}$ such that for all $q\in\mathcal{Q}$, and $n\in\{0,...,N\}$:

[TABLE]

The general terms of the sequence $(k_{n,s})$ are given for $s\in\mathbb{R}_{+}^{*}$ by:

[TABLE]

Proof.

For $s=0$ we take by convention $k_{n,0}=0$ for all $n\in\{0,...,N\}$. Let $q\in\mathcal{Q}^{*}$, we proceed by backward induction on the variable $n$:

For $n=N$, $\Psi_{N}(1,q)=0$ for any $q\in\mathcal{Q}^{*}$ and thus, one can take $k_{N,s}=0$ and $c_{N,s}=0$ for all $s>0$. Assume the result is true for $n=m$, i.e., for all $q\in\mathcal{Q}^{*}$, there exist $k_{m,s}\in\mathcal{M}^{I\times J}$ and $c_{m,s}\in\mathbb{R}$, such that (2.16) is satisfied. For $n=m$, for all $q\in\mathcal{Q}^{*}$, we get from (2.15):

[TABLE]

and thus (2.16) is satisfied if we put:

[TABLE]

This ends the induction. ∎

Notation. Given $N\in\mathbb{N}^{*}$ and $q_{0}\in\mathcal{Q}$, for all $t\in\mathcal{P}_{N}$, we define the subset of $\mathcal{Q}$:

[TABLE]

Proposition 2.5**.**

Given $N\in\mathbb{N}^{*}$ and $q_{0}\in\mathcal{Q}$, for all $t\in\mathcal{P}_{N}$ and all $q\in\mathcal{Q}_{N}(t,q_{0})$,

[TABLE]

Proof.

Both functions share the same terminal condition, i.e $\Psi_{N}(1,q)=W_{\mathcal{P}_{N}}(1,q)=0$, for all $q\in\mathcal{Q}$, (see (2.1), and characterization of $W_{\mathcal{P}_{N}}$ in terms of discrete DPP). Thus, it suffices to prove that $\Psi_{N}$ and $W_{\mathcal{P}_{N}}$ satisfy the same recursive formula. To that purpose, fix $q_{0}\in\mathcal{Q}$ and time $t=\frac{k}{N}$, where k\in\big{\{}0,...,N-1\big{\}}. By the discrete DPP, it follows that for all $q\in\mathcal{Q}_{N}\left(\frac{k}{N},q_{0}\right)$,

[TABLE]

By (2.15), for any $k\in\left\{0,...,N-1\right\}$,

[TABLE]

where $q\in\mathcal{Q}_{N}\left(\frac{k}{N},q_{0}\right)$. Equivalently:

[TABLE]

By Proposition 2.4, $\Psi_{N}$ is affine in the state variable $q$ and it thus follows:

[TABLE]

Hence, due to $\sum_{ij}u_{i}v_{j}=1$, we get:

[TABLE]

This in view of (2.17), proves that $\Psi_{N}=W_{\mathcal{P}_{N}}$. ∎

2.5 Coincidence of $\Psi_{\lambda}$ and $W_{\mathcal{P}_{\lambda}}$

In view of Contou-Carrère (2011), the convergence is not uniform in the state variable and as a consequence we cannot use the Tauberian theorem of Ziliotto (2016) to obtain an immediate result on the convergence of the $\lambda$-discounted value, when $\lambda$ tends to [math]. In the case of the $n$-stage value, coincidence between $\Psi_{N}$ and $W_{\mathcal{P}_{N}}$ follows immediately from the preceding paragraph since both functions admit the terminal value zero and satisfy the same recursive equation. Concerning the $\lambda$-discounted game, we need to follow a slightly different approach to prove such a coincidence.

Recall that the map $\mathbf{V}_{\lambda}:\mathcal{Z}\rightarrow\mathbb{R}$ has been characterised by the recursive formula (1.2). It is then clear that an extension of $\textbf{V}_{\lambda}$ to a map $\textbf{V}_{\lambda}:\mathcal{Q}\rightarrow\mathbb{R}$ is obtained if we define it by the same recursive formula; namely, for any $q\in\mathcal{Q}$,

[TABLE]

Given $\lambda\in(0,1)$, let us define the function $\Psi_{\lambda}:\mathcal{Q}\rightarrow\mathbb{R}$, such that

[TABLE]

By (2.18), $\Psi_{\lambda}$ satisfies the following equation:

[TABLE]

By 2.14, for any $q\in\mathcal{Q}$, we get:

[TABLE]

where the last equation follows by stationarity of the discounted game $\mathcal{G}_{\mathcal{P}_{\lambda}}(t,q)$. In the sequel, we put $W_{\mathcal{P}_{\lambda}}(q)=W_{\mathcal{P}_{\lambda}}(0,q)$.

Notation. We use the following notations:

• •

The norm $\left\|\cdot\right\|_{\infty}$ of a bounded real-valued function $f$, defined on $\mathcal{Q}$, is

[TABLE]

• •

$\mathcal{F}_{\mathcal{B}}$ stands for the set of real-valued bounded functions $f$ defined on $\mathcal{Q}$ with the norm $\left\|\cdot\right\|_{\infty}$. Clearly, $\mathcal{F}_{\mathcal{B}}$ is a Banach space.

• •

$\mathcal{L}_{\mathcal{B}}$ stands for the subspace of $\mathcal{F}_{\mathcal{B}}$, such that if $f\in\mathcal{L}_{\mathcal{B}}$ then there exist bounded and measurable $K:\mathbb{R}\rightarrow\mathbb{R}^{I\times J}$ and $c:\mathbb{R}\rightarrow\mathbb{R}$, such that $f(q)=\left\langle K(|q|),q\right\rangle+c(|q|)$, for any $q\in\mathcal{Q}$.

• •

For all $f\in\mathcal{F}_{\mathcal{B}}$ and any $\lambda\in(0,1)$, we define the operator $\Theta_{\lambda}$ as follows:

[TABLE]

and clearly $\Theta_{\lambda}$ admits $W_{\mathcal{P}_{\lambda}}(\cdot)$ as unique fixed point.

• •

For all $f\in\mathcal{F}_{\mathcal{B}}$ and any $\lambda\in(0,1)$, we define the operator $\textbf{T}_{\lambda}$ as follows:

[TABLE]

Likewise, $\textbf{T}_{\lambda}$ admits $\Psi_{\lambda}(\cdot)$ as unique fixed point.

• •

Given $\lambda\in(0,1]$, for any $t\in\mathcal{P}_{\lambda}$, we define the following subset of $\mathcal{Q}$:

[TABLE]

Proposition 2.6**.**

If $f\in\mathcal{L}_{\mathcal{B}}$, then $\textbf{T}_{\lambda}(f)\in\mathcal{L}_{\mathcal{B}}$ and $\textbf{T}_{\lambda}(f)=\Theta_{\lambda}(f)$.

Proof.

Let $f\in\mathcal{L}_{\mathcal{B}}$. By definition of the operator $\textbf{T}_{\lambda}$, it is easy to see that $\textbf{T}_{\lambda}(f)$ is also bounded. For the rest of the proof, let $\lambda\in(0,1)$ and fix $t\in\mathcal{P}_{\lambda}$. Then, for any $q\in\mathcal{Q}_{\lambda}(t)$,

[TABLE]

Hence,

[TABLE]

[TABLE]

It is easy to see that there exist $K^{\prime}:\mathbb{R}\rightarrow\mathbb{R}^{I\times J}$ and $c^{\prime}:\mathbb{R}\rightarrow\mathbb{R}$ such that $\textbf{T}_{\lambda}(f)(q)=\left<K^{\prime}(|q|),q\right>+c^{\prime}(|q|)$, where

[TABLE]

and it thus, clearly follows that $\textbf{T}_{\lambda}(f)\in\mathcal{L}_{\mathcal{B}}$.

For the rest of the proof, since $f\in\mathcal{L}_{\mathcal{B}}$, by equation (2.22) for any $q\in\mathcal{Q}$,

[TABLE]

which concludes the proof. ∎

Corollary 2.7**.**

Fix $\lambda\in(0,1]$. Then, for any $q\in\mathcal{Q}$, we have $\Psi_{\lambda}(q)=W_{\mathcal{P}_{\lambda}}(q)$.

Proof.

By Proposition 2.6, since $f\equiv 0\in\mathcal{L}_{\mathcal{B}}$, we get that for any $m\in\mathbb{N}^{*}$, $\textbf{T}^{m}(0)=\Theta^{m}(0)$. It is easy to see that $\textbf{T}_{\lambda}$ and $\Theta_{\lambda}$ are contracting operators and it thus follows $\lim_{m\rightarrow+\infty}\textbf{T}^{m}_{\lambda}(0)=\lim_{m\rightarrow+\infty}\Theta^{m}_{\lambda}(0)$. In view of equations (2.22) and (2.23), we conclude that $\Psi_{\lambda}=W_{\mathcal{P}_{\lambda}}$. ∎

This result will allow us to use approximation schemes for differential games in the subsequent parts of the proof. Since the value of the original game is equal to that of the discretized approximated game, proving convergence of the value of the latter will prove the convergence of the value of the former. One difficulty arises however from the irregularity of the differential game at the origin.

3 Existence of the limit value in $\Gamma_{N}(z)$

In this section, we prove the main result of the paper. We show that the asymptotic values of the $N$-stage and the $\lambda$-discounted games exist and they are independent of the initial state $z$. We further show that $\lim_{N\rightarrow+\infty}\textbf{V}_{N}(z)=\lim_{\lambda\rightarrow 0}\textbf{V}_{\lambda}(z)=W(0,0)$. For this purpose we first provide some useful lemmas on the value of the original game $\textbf{V}_{N}$ (resp. $\textbf{V}_{\lambda}$) and the associated function $\Psi_{N}$ (resp. $\Psi_{\lambda}$).

Lemma 3.1**.**

Let $\omega=(i_{t},j_{t})_{t\in\mathbb{N}^{*}}$ be a play and $(z_{t})_{t\in\mathbb{N}}$ be the process in $\mathcal{Z}$, induced by the initial position $z_{0}=0$ and the play $\omega$. Then, for any $z\in\mathcal{Z}^{*}$, we have:

• •

for any $N\in\mathbb{N}^{*}$,

[TABLE]

where $C=2\sup_{N\in\mathbb{N}^{*}}\varepsilon(N)$ and $\varepsilon(N)$ is a function which goes to zero when $N$ tends to infinity.

• •

for any $\lambda\in(0,1]$, such that $\lambda z<1$, there exists $N\in\mathbb{N}^{*}$, i.e., $N+|z|=\lfloor{\frac{1}{\lambda}}\rfloor$, such that we have:

[TABLE]

where $C^{z}_{\lambda}=(1-\lambda)^{\lfloor{\frac{1}{\lambda}}\rfloor-|z|}+C$.

Proof.

Fix $z\in\mathcal{Z}$ and $N\in\mathbb{N}^{*}$. Then, we have:

[TABLE]

It follows,

[TABLE]

where $\varepsilon(x)$ is a function that goes to 0 when $x$ tends to infinity. We put $C:=2\sup_{N\in\mathbb{N}^{*}}\varepsilon(N)$ and we thus, conclude the proof of the assertion. To show the second assertion, in a similar way, we obtain:

[TABLE]

Define $N:=\inf\{k\in\mathbb{N}^{*}:k+|z|+1>1/\lambda\}$. Hence, we may write:

[TABLE]

which ends the proof of the Lemma. ∎

Proposition 3.2**.**

Let $z\in\mathcal{Z}$. If $\textbf{V}_{N}(z)$ converges to some $\ell\in\mathbb{R}$ (resp. $\textbf{V}_{\lambda}(z)$), then for any $\tilde{z}\in\mathcal{Z}$, $\textbf{V}_{N}(\tilde{z})$ (resp. $\textbf{V}_{\lambda}(\tilde{z})$) converges to the same limit $\ell$.

Proof.

Given $N\in\mathbb{N}^{*}$ ($\lambda\in(0,1)$), fix $z\in\mathcal{Z}^{*}$ and let us consider the games $\Gamma_{N}(z)$ (resp. $\Gamma_{\lambda}(z)$) and $\Gamma_{N}(0)$ (resp. $\Gamma_{\lambda}(0)$). For a pair of behavioral strategies $(\sigma,\tau)$ we denote by $\mathbb{P}_{\sigma,\tau}$ the probability induced on $(I\times J)^{N}$, (resp. ($I\times J)^{\infty}$). With respect to the probability $\mathbb{P}_{\sigma,\tau}$, by Lemma 3.1, we get:

[TABLE]

Since the right hand term is independent of $(\sigma,\tau)$ and the sup-norm of the value function is less than or equal to sup-norm of the payoff function, we get:

[TABLE]

The conclusion follows by remarking that the right hand side goes to zero when $N\rightarrow\infty$.

For the rest of the proof, fix $\lambda\in(0,1)$. Likewise, by Lemma 3.1, we get:

[TABLE]

Likewise, we obtain:

[TABLE]

Note that $C_{\lambda}^{z}$ converges to $1/e$, as $\lambda$ tends to [math]. The result follows since the right hand side goes to zero when $\lambda$ tends to zero. ∎

Next theorem provides the main result of the paper. Given $z\in\mathcal{Z}$, we first show that $\textbf{V}_{N}(z)$ converges to $W(0,0)$ when $N$ tends to infinity and we then prove that $\textbf{V}_{\lambda}(z)$ converges to the same limit, when $\lambda$ goes to 0.

Theorem 3.3**.**

For any $z\in\mathcal{Z}$, $\lim\limits_{N\rightarrow+\infty}\textbf{V}_{N}(z)=\lim\limits_{\lambda\rightarrow 0}\textbf{V}_{\lambda}(z)=W(0,0)$.

Proof.

We first prove that for any $z\in\mathcal{Z}$, $\lim_{N\rightarrow+\infty}\textbf{V}_{N}(z)=W(0,0)$. To that purpose, fix $\varepsilon>0$, choose $\eta\in(0,\frac{1}{4})$ such that,

[TABLE]

and in view of Theorem 2.3, we also require $\eta$ to be such that for all $q\in\mathcal{Q}^{*}$ with $\left|q\right|\leq\eta$, $\left|W\left(0,q\right)-W(0,0)\right|<\frac{\varepsilon}{3}$. Choose some $q_{0}$ such that $|q_{0}|=\eta$. Assumptions on the strategy sets, the dynamics and running payoff functions of Theorem 4.4 in Souganidis (1999) are established in \mathcal{G}\big{(}0,q_{0}\big{)}. Accordingly, there exists $\delta>0$, such that for all $|\mathcal{P}|<\delta$, the value $W_{\mathcal{P}}$ converges uniformly on every compact set of $\mathcal{Q}$ to $W$, as the mesh of the discretization $|\mathcal{P}|$ tends to [math]. Fix $N_{0}=\lfloor\frac{1}{\delta}\rfloor+1$ and associate to \mathcal{G}\big{(}0,q_{0}\big{)}, for all $N\geq N_{0}$, a discrete time game adapted to the subdivision $\mathcal{P}_{N}$, denoted by \mathcal{G}_{\mathcal{P}_{N}}\big{(}0,q_{0}\big{)}. Then, \big{|}W_{\mathcal{P}_{N}}\left(0,q_{0}\right)-W\left(0,q_{0}\right)\big{|}<\frac{\varepsilon}{3}. From Proposition 2.5, $W_{\mathcal{P}_{N}}\left(0,q_{0}\right)=\Psi_{N}(0,q_{0})$. By Lemma 3.1, for any $z\in\mathcal{Z}$, we have:

[TABLE]

There exists $z_{0}\in\mathcal{Z}$, such that $z_{0}=\lfloor Nq_{0}\rfloor$ and since $|q_{0}|=\eta$, we have $|z_{0}|=N\eta-\rho$ for some $\rho\in(0,1)$. By definition $\Psi_{N}(0,q_{0})=\textbf{V}_{N}(\lfloor Nq_{0}\rfloor)$. Hence,

[TABLE]

Since $\eta<\frac{1}{4}$, we have $\ln(1+\eta-\rho/N)<\ln(2)$ and $(\eta-\rho/N)|\ln(\eta-\rho/N)|<\eta|\ln(\eta)|$. As a consequence, we get:

[TABLE]

Therefore, for every integer $N\geq N_{0}$,

[TABLE]

From (2.1), $\Psi_{N}\left(0,0\right)=\textbf{V}_{N}(0)$. It follows that $\textbf{V}_{N}(0)\to W(0,0)$ when $N\to\infty$. In view of Lemma 3.2, we conclude that for any $z\in\mathcal{Z}$, $\textbf{V}_{N}(z)$ converges to $W(0,0)$ as $N$ tends to infinity.

To prove the assertion on the discounted value, likewise we fix $\varepsilon>0$ and choose $\eta>0$ such that we have:

[TABLE]

In view of Theorem 2.3, we also require $\eta$ to be such that for all $q\in\mathcal{Q}^{*}$ with $\left|q\right|\leq\eta$, $\left|W\left(0,q\right)-W(0,0)\right|<\frac{\varepsilon}{3}$. Following similar arguments to the ones of the first part of the proof, fix $\lambda_{0}:=\delta$ and associate to \mathcal{G}\big{(}0,q_{0}\big{)}, for all $\lambda\leq\min(\lambda_{0},\eta)$, a discrete time game adapted to the subdivision $\mathcal{P}_{\lambda}$, denoted by \mathcal{G}_{\mathcal{P}_{\lambda}}\big{(}0,q_{0}\big{)}. Then, $\left|W_{\mathcal{P}_{\lambda}}\left(q_{0}\right)-W\left(0,q_{0}\right)\right|<\frac{\varepsilon}{3}$. By Corollary 2.7, $W_{\mathcal{P}_{\lambda}}\left(q_{0}\right)=\Psi_{\lambda}(q_{0})$. By Lemma 3.1, for any $z\in\mathcal{Z}$, we have:

[TABLE]

There exists $z_{0}\in\mathcal{Z}$, such that $z_{0}=\lfloor\frac{q_{0}}{\lambda}\rfloor$ and since $|q_{0}|=\eta$, we have: $|z_{0}|=(\eta/\lambda)-\rho$ for some $\rho\in(0,1)$. By definition $\Psi_{\lambda}(q_{0})=\textbf{V}_{\lambda}\left(\lfloor\frac{q_{0}}{\lambda}\rfloor\right)$. It follows:

[TABLE]

Since $\eta\in(0,\frac{1}{4})$, we have $(\eta-\lambda\rho)|\ln(\eta-\lambda\rho)|<\eta|\ln(\eta)|$. It then follows:

[TABLE]

Therefore, for any $\lambda\leq\min\{\lambda_{0},\eta\}$,

[TABLE]

From (2.19), $\Psi_{\lambda}\left(0\right)=\textbf{V}_{\lambda}(0)$. It follows that $\textbf{V}_{\lambda}(0)\to W(0,0)$ when $\lambda\to 0$. By Lemma 3.1, we conclude that for any $z\in\mathcal{Z}$, $\textbf{V}_{\lambda}(z)$ converges to $W(0,0)$ as $\lambda$ tends to zero, which completes the proof of the Theorem. ∎

4 Conclusion and perspectives

In this paper we have studied two-player zero-sum frequency-dependent games with separable stage-payoffs and established the convergence of $\textbf{V}_{n}$ and $\textbf{V}_{\lambda}$ as $n$ tends to infinity and $\lambda$ goes to [math] respectively, to the value of the associated differential game starting at the origin, $W(0,0)$. A natural generalization of our existence result concerns a stage payoff function $g(z,i,j)$ which is assumed to be linear in $z$ and such that the impacts of the past and that of present actions are not separable, but combine together in some way. Lastly, let us mention that since existence of the asymptotic value in the zero-sum case is established, a study of limits of Nash equilibria payoffs in general-sum frequency-dependent games that leads to some Folk-Theorem-like now seems to be possible. In doing so, one may compare the asymptotic results with the ones obtained for infinite games by Joosten et al. (2003).