Acyclic Gambling Games

TL;DR

This paper studies the limiting behavior of two-player zero-sum stochastic gambling games with controlled wealth states, establishing conditions for convergence of discounted values to a unique solution characterized by functional equations.

Contribution

It introduces a strong acyclicity condition ensuring the existence and characterization of the limit of discounted game values as the discount factor approaches zero.

Findings

Limit exists under strong acyclicity condition.

Limit characterized as a unique solution to functional equations.

Counterexample shows convergence can fail under weaker conditions.

Abstract

We consider 2-player zero-sum stochastic games where each player controls his own state variable living in a compact metric space. The terminology comes from gambling problems where the state of a player represents its wealth in a casino. Under natural assumptions (such as continuous running payoff and non expansive transitions), we consider for each discount factor the value v $\lambda$ of the $\lambda$-discounted stochastic game and investigate its limit when $\lambda$ goes to 0. We show that under a strong acyclicity condition, the limit exists and is characterized as the unique solution of a system of functional equations: the limit is the unique continuous excessive and depressive function such that each player, if his opponent does not move, can reach the zone when the current payoff is at least as good than the limit value, without degrading the limit value. The approach generalizes and provides a new viewpoint on the Mertens-Zamir system coming from the study of zero-sum repeated games with lack of information on both sides. A counterexample shows that under a slightly weaker notion of acyclicity, convergence of (v $\lambda$) may fail.

Equations

$${\Phi }_w(x)=\{w{\delta }_{\min \{x+s,1\}}+(1-w){\delta }_{x-s}:0\le s\le x\}.$$

$$v_{\lambda }(x,y)$$

$$v_{n+1}(x,y)$$

$$\forall x\in X,\forall x^{\prime }\in X,\forall p\in \Gamma (x),\exists p^{\prime }\in \Gamma (x^{\prime }),s.t.d_{KR}(p,p^{\prime })\le d(x,x^{\prime }),$$

$$andsimilarly:\forall y\in Y,\forall y^{\prime }\in Y,\forall q\in \Lambda (y),\exists q^{\prime }\in \Lambda (y^{\prime }),s.t.d_{KR}(q,q^{\prime })\le d(y,y^{\prime }).$$

$$\forall x\in X,{\mathrm{Argmax}}_{p\in \Gamma (x)}\tilde{\phi }(p)=\{{\delta }_x\}.$$

$$\forall y\in Y,{\mathrm{Argmin}}_{q\in \Lambda (y)}\tilde{\psi }(q)=\{{\delta }_y\}.$$

$$\forall x\in X,{\mathrm{Argmax}}_{p\in {\Gamma }^{\infty }(x)}\tilde{\phi }(p)=\{{\delta }_x\}.$$

$$\forall y\in Y,{\mathrm{Argmin}}_{q\in {\Lambda }^{\infty }(y)}\tilde{\psi }(q)=\{{\delta }_y\}.$$

$$v(a)=v(b)=1,v(c)=0.$$

$$X=\{1-\frac{1}{n},n\in I{N}^{\ast }\}\cup \{1\}=\{x_1,...,x_n,....,x_{\infty }\},$$

$$\Gamma (x_n)=\{\alpha {\delta }_{x_n}+(1-\alpha ){\delta }_{x_{n+1}},\alpha \in [0,1]\},\mathrm{and}\Gamma (x_{\infty })=\{{\delta }_{x_{\infty }}\}.$$

$$\left(\begin{array}[]{cc}w({n+1,m})&w({n+1,m+1})\\ w({n,m})&w({n,m+1})\\ \end{array}\right).$$

$$\left(\begin{array}[]{cc}w({n+2,m})&w({n+2,m+1})\\ w({n+1,m})&w({n+1,m+1})\\ \end{array}\right).$$

$$w(n+2,m+1)-w(n+1,m+1)>w(n+1,m)-w(n,m).$$

$$w(n+p+1,m+p)\ge w(n+1,m)-w(n,m)>0.$$

$$h(x)=\mathrm{Max}\{v(p,y)-v(x,y),y\in Y,p\in \Gamma (x)\}.$$

$$v(p_2,q_1)-v(p_1,q_1)\ge v(p_1,y_0)-v(x_0,y_0)=h_0.$$

$$p_{t+1}^n\in {\mathrm{Argmax}}_{p\in \tilde{\Gamma }(p_t^n)}\left({\lambda }_{n}u(p_t^n,y)+(1-{\lambda }_n)v_{{\lambda }_n}(p,y)\right).$$

$${\lambda }_{n}u(p_t^n,y)+(1-{\lambda }_n)v_{{\lambda }_n}(p_{t+1}^n,y)\ge v_{{\lambda }_n}(p_t^n,y).$$

$$v_{{\lambda }_n}(p_{t+1}^n,y)\ge v_{{\lambda }_n}(p_t^n,y)+\frac{{\lambda }_n^2}{1-{\lambda }_n}>v_{{\lambda }_n}(p_t^n,y).$$

$$v_{{\lambda }_n}(x,y)\le v_{{\lambda }_n}(p_{T_n-1}^n,y)\le {\lambda }_{n}u(p_{T_n-1}^n,y)+(1-{\lambda }_n)v_{{\lambda }_n}(p_{T_n}^n,y).$$

$$v(p^{\ast },y)\le u(p^{\ast },y).$$

$$D{=}_{def}\{(p^{\prime },p^{\prime \prime })\in \Delta (X)\times \Delta (X),h(p^{\prime \prime })\le h(p^{\prime })\}.$$

$$Z={\mathrm{Argmax}}_{(x,y)\in X\times Y}{v}_1(x,y)-v_2(x,y).$$

$$u(x,y)=0\mathrm{if}x=y,u(x,y)=1\mathrm{if}x\ne y.$$

$$a^{\prime }$$

$$v(c,a^{\prime })\le {\beta }^{\ast }v(c,b^{\prime })+{{\beta }^{\ast }}^{2}v(c,c^{\prime })+(1-{\beta }^{\ast }-{{\beta }^{\ast }}^2)v(c,a^{\prime }).$$

$$v(a,a^{\prime })\ge \alpha v(b,a^{\prime })+{\alpha }^{2}v(c,a^{\prime })+(1-\alpha -{\alpha }^2)v(a,a^{\prime }).$$

$$a^{\prime }$$

Full text

1 Introduction

The model of zero-sum stochastic games was introduced by Shapley [29] in 1953. A state variable $\omega\in\Omega$ follows a controlled Markov chain with transitions $Q(\tilde{\omega}|i,j,\omega)$ controlled by the actions of two competing players ($i\in I$ for player 1 and $j\in J$ for player 2). Shapley assumed the action and state spaces ($I$, $J$ and $\Omega$) to be finite and proved the existence of the value $v_{\lambda}$ of the $\lambda$-discounted game using a dynamic programming principle, and characterized $v_{\lambda}$ as the unique fixed point of what has been called the Shapley operator [28, 31].

Bewley and Kohlberg [3], using algebraic tools, proved the existence of the asymptotic value $v={\rm lim}_{\lambda\rightarrow 0}v_{\lambda}$. Actually, when action and state spaces are finite, the equations that define $v_{\lambda}$ may be described by finitely many polynomial inequalities, implying that $v_{\lambda}$ is semi-algebraic and so is converging. The extension of this result to infinite stochastic games is a central question in mathematical game theory [16, 22, 24, 31].

Recently, several important conjectures [20, 22] were proved to be false. Vigeral [33] and Ziliotto [34] provided examples where the family $\{v_{\lambda}\}$ diverges as $\lambda$ approaches zero. In Vigeral, the state space $\Omega$ is finite and the action sets $I$ and $J$ are semi-algebraic. In Ziliotto, the set of actions is finite but the state space $\Omega$ is compact, and can be seen as the space of common beliefs on a finite state variable, controlled but not observed by the players.

On the other hand, there are many classes of stochastic games with general state space and action sets where $\{v_{\lambda}\}$ converges (for a recent survey, see [16]). Many have in common some irreversibility in the transitions. In recursive games [8, 32] the current payoff is zero until the game is absorbed. In absorbing games [12, 21, 28] there is only one non-absorbing state. In repeated games with incomplete information [2, 23, 28], once a player reveals some information, he cannot withdraw it. Similarly in splitting games [14, 15, 31] the state follows a martingale which eventually converges. Interestingly, in all those classes of “irreversible” stochastic games, not only we have convergence but also an explicit characterization of the asymptotic value. This leads many to anticipate that irreversibility has to do with convergence.

Our paper provides a weak and a strong definition of irreversibility (we call acyclicity) and prove that they constitute the frontier between convergence and divergence of $\{v_{\lambda}\}$: strong acyclicity guarantee convergence while the closely related weak acyclicity do not. To do so, we restrict ourself to a new class which embeds any product stochastic game [9] and naturally extends gambling houses.

A classical gambling house problem [6, 19] has three ingredients : a metric state space $S$, a Borel-measurable utility function $u:S\rightarrow I\!\!R$, and a gambling house $\Phi$, where $\Phi$ is a set value function that assigns to each $s\in S$ a set $\Phi(s)$ of $\Delta(S)$ (set of Borel probability distributions over $S$). At each stage $t$, given the state $s_{t}$, the decision maker gets the reward $u(s_{t})$, chooses $p_{t}\in\Phi(s_{t})$, and the state moves to $s_{t+1}$ according to the probability distribution $p_{t}$.

The gambling house is leavable if for every $s\in S$, $\delta_{s}$ (the dirac mass at $s$) belongs to $\Phi(s)$. It is well known that any MDP can be mapped to a gambling house and any positive MDP to a leavable gambling house [7, 19, 30]. In this paper, we consider only leavable gambling houses.

In a gambling game, each player controls his gambling house: $\Gamma:X\rightarrow\Delta(X)$ for Player 1 and $\Lambda:Y\rightarrow\Delta(Y)$ for Player 2, and the utility function is now $u:X\times Y\rightarrow I\!\!R$ with the convention that player 1 wants to maximize $u$ whereas player 2 wants to minimize $u$. At each stage $t$, both players knowing the state $\omega_{t}=(x_{t},y_{t})$, simultaneously Player 1 chooses $p_{t}$ in $\Gamma(x_{t})$ and Player 2 chooses $q_{t}\in\Lambda(y_{t})$, the stage payoff is $u(x_{t},y_{t})$ and a new state $(x_{t+1},y_{t+1})$ is selected according to the probability distribution $p_{t}\otimes q_{t}$. As any MDP can be mapped into a gambling house, a product stochastic game [9] can be mapped into a gambling game.

For each $\lambda\in(0,1]$, one can define the $\lambda$-discounted game where the stream of payoffs is evaluated according to $\sum_{t=1}\lambda(1-\lambda)^{t-1}u(x_{t},y_{t})$. Usually $\lambda$ is called the discount rate, $1-\lambda=\frac{1}{1+r}$ is called the discount factor and $r$ is the interest rate. Under classical regularity assumptions, the $\lambda$-discounted game has a value $v_{\lambda}$ and the family $\{v_{\lambda}\}$ is equi-continuous.

Our first main result shows that if at least one of the gambling houses $\Gamma$ or $\Lambda$ is strongly acyclic, $\{v_{\lambda}\}$ uniformly converges to a function $v$ as $\lambda$ goes to [math]. Moreover, we provide several characterizations of the asymptotic value $v$ that extend the well known Mertens-Zamir system of functional equations [23]. Our second result proves that under a slightly weaker notion of acyclicity, $\{v_{\lambda}\}$ may diverge (even if both houses $\Gamma$ and $\Lambda$ are weakly acyclic and both state spaces $X$ and $Y$ are finite). Our example is inspired by an example in Ziliotto [34] for stochastic games where both players control the same state variable. It is the first in the class of product stochastic games, and appears simpler than the existing counterexamples of divergence with finite state space. Finally, under an idempotent assumption combined with a bounded variation hypothesis on the transitions, we prove existence of the uniform value, extending a recent result by Oliu-Barton [25] on splitting games.

2 Gambling Games

2.1 Notations

Given a compact metric space $S$, we denote by $\mathcal{B}(S)$, resp. by $\mathcal{C}(S)$, the set of bounded measurable, resp. continuous, functions from $S$ to the reals, and by $\Delta(S)$ the set of Borel probabilities over $S$. For $s$ in $S$, we denote by $\delta_{s}\in\Delta(S)$ the Dirac measure on $s$, and whenever possible we assimilate $s$ and $\delta_{s}$. For $v$ in $B(S)$, we denote by $\tilde{v}$ its affine extension to $\Delta(S)$: $\tilde{v}(p)=I\!\!E_{p}(v)$ for all $p$ in $\Delta(S)$, where $I\!\!E_{p}(v)$ is the expectation of $v$ with respect to $p$. $\Delta(S)$ is endowed with the weak-* topology, a compatible distance being the Kantorovich-Rubinstein (or Wasserstein of order 1) metric: $d_{KR}(p,p^{\prime})=\sup_{v\in E_{1}}|\tilde{v}(p)-\tilde{v}(p^{\prime})|$, where $E_{1}$ is the set of 1-Lipschitz functions on $S$. When there is no confusion, $\tilde{v}(p)$ will also be denoted by $v(p)$.

2.2 Model

A gambling game is a zero-sum stochastic game where each player controls his own state variable. We will always assume in this paper that the state spaces are non empty metric and compact, and denote by $X$ and $Y$ the respective set of states111Both metrics are denoted by $d$, and we will use the metric $d((x,y),(x^{\prime},y^{\prime}))=d(x,x^{\prime})+d(y,y^{\prime})$ on $X\times Y$. controlled by Player 1 and by Player 2. The transitions of Player 1 are given by a continuous222i.e. $\forall\varepsilon>0,\exists\alpha>0,\forall x,x^{\prime}\in X\;{\rm with}\;d(x,x^{\prime})\leq\alpha,\forall p\in\Gamma(x),\exists p^{\prime}\in\Gamma(x^{\prime})\;{\rm s.t.}\;d_{KR}(p,p^{\prime})\leq\varepsilon$. Since $\Gamma$ has compact values, it implies that the graph of $\Gamma$ is compact. multifunction $\Gamma:X\rightrightarrows\Delta(X)$ with non empty convex333If originally the gambling game has non convex values, then allowing as usual players to randomize, would lead to transitions with convex values. compact values: if the state of Player 1 is at $x$, he can select his new state according to any probability in $\Gamma(x)$. Similarly, a continuous multifunction $\Lambda:Y\rightrightarrows\Delta(Y)$ with non empty convex compact values, gives the transitions of Player 2. The players independently control their own state, and only interact through payoffs: the running payoff of Player 1 is given by a continuous mapping $u:X\times Y\longrightarrow I\!\!R$, and the payoff to Player 2 is given by $-u$.

Gambling games extend the model of gambling houses [6], which correspond to the single player case when $Y$ is a singleton and Player 2 plays no role. One can show, by an adequate increase of the state space in order to encompass actions, that any MDP can be mapped into a gambling house [7, 19, 30].

A standard gambling house is the red-and-black casino where $X=[0,1]$ is a fortune space. Suppose that at each fortune $x\geq 0$, the gambler can stake any amount $s$ in her possession. The gambler loses the stake with probability $1-w$ where $w\in(0,1)$ is fixed and given, and wins back the stake and an additional equal amount with probability $w$. The corresponding transition multiifunction reads:

[TABLE]

Another class of gambling house are splitting problems where $X=\Delta(K)$ is a simplex ($K$ is a finite set) and $\Gamma(x)$ is the set of Borel probabilities on $X$ centered at $x$. The idea of splitting was introduced by Aumann and Maschler [2] in the context of repeated games with incomplete information on one side. This is now very popular in economics (persuasion and information design literature [13]).

The above examples of gambling houses naturally extend to gambling games. One can consider a casino game where each player $i$ controls a red-and-black house with parameter $w_{i}$, and the running payoff depends on the current pair of fortunes. Another example is a splitting game [14, 15, 31] where $X=\Delta(K)$ and $Y=\Delta(L)$ are simplexes, $\Gamma(x)$ is the set of Borel probability measures on $X$ that are centered at $x$ and $\Lambda(y)$ is the set of probability measures of Borel probability measures on $Y$ that are centered at $y$.

2.3 Discounted Evaluations

Given a discount factor $\lambda\in(0,1]$ and an initial state $(x_{1},y_{1})$ in $X\times Y$, the game $G_{\lambda}(x_{1},y_{1})$ is played as follows: at any stage $t\geq 1$, the payoff to Player 1 is $u(x_{t},y_{t})$ and both players knowing $(x_{t},y_{t})$, simultaneously Player 1 chooses $p_{t+1}$ in $\Gamma(x_{t})$ and Player 2 chooses $q_{t+1}$ in $\Lambda(y_{t})$. Then, $x_{t+1}$ and $y_{t+1}$ are independently selected according to $p_{t}$ and $q_{t}$, the new states $x_{t+1}$ and $y_{t+1}$ are publicly announced, and the play goes to stage $t+1$. Under our assumptions of compact state spaces, continuous transitions with convex compact values and continuous running payoff, it is easy to describe the value of such dynamic game.

Definition 2.1**.**

$v_{\lambda}$* is the unique element of $\mathcal{C}(X\times Y)$ s.t. $\forall(x,y)\in X\times Y,$*

[TABLE]

This is the standard characterization of the value of a discounted game by means of the Shapley operator. Existence and uniqueness of $v_{\lambda}$ follow from standard fixed-point arguments (see for instance [22, 28]). We refer to $v_{\lambda}(x,y)$ as the value of the game $G_{\lambda}(x,y)$.

The goal of the paper is to study the convergence of $(v_{\lambda})_{\lambda}$ when $\lambda$ goes to 0, i.e. when players become more and more patient.

Remark 2.2**.**

Cesaro Evaluations. It is also standard to define the value of the $n$-stage games by: $v_{1}=u$, and for $n\geq 1$ and $(x,y)\in X\times Y$:

[TABLE]

It is known that the uniform convergence of $(v_{n})_{n}$ when $n$ goes to infinity, is equivalent to the uniform convergence of $(v_{\lambda})_{\lambda}$ when $\lambda$ goes to 0, and in case of convergence both limits are the same (Theorem 2.2 in [35] applies here)

2.4 Non expansive transitions

Without further assumptions, convergence of $(v_{\lambda})$ may fail even in the simple case where $\Gamma$ and $\Lambda$ are single-valued (“0 player case”, players have no choice), so we will assume throughout the paper that the gambling game is non expansive, i.e. have non expansive transitions:

Definition 2.3**.**

The game has non expansive transitions if:

[TABLE]

[TABLE]

The gambling game has non expansive transitions if the transitions, viewed as mappings from $X$ to $2^{\Delta(X)}$, and from $Y$ to $2^{\Delta(Y)}$, are 1-Lipschitz for the Hausdorff distance on compact subsets of ${\Delta(X)}$ and ${\Delta(Y)}$. Note that the transitions are always non expansive when $X$ and $Y$ are finite444If $X$ is finite and $d(x,x^{\prime})=2$ for $x\neq x^{\prime}$, then $d_{KR}(p,p^{\prime})=\|p-p^{\prime}\|_{1}$ for $p$, $p^{\prime}$ in $\Delta(X)$.. Moreover splitting games are non expansive [14], and red-and-black casino houses with parameter $w$ are non-expansive if and only if $w\leq\frac{1}{2}$ [17]. Let us mention also Markov chain repeated games with incomplete information [11], where each player observes a private and exogenous Markov chain. These repeated games lead to gambling houses with transitions of the form: $X$ is a simplex $\Delta(K)$, and $\Gamma(x)=\{pM,p\in\Delta(X)\;{\rm centered\;at\;}x\}$ with $M$ a fixed stochastic matrix. Here again, transitions are non expansive.

Let us mention immediately an important consequence of the non expansive assumption. The proof is in the Appendix.

Proposition 2.4**.**

Assume the gambling game is non-expansive. Then the family $\{v_{\lambda}\}_{\lambda\in(0,1]}$ is equicontinuous.

This proposition extends to two players a similar result in [17] on gambling houses where it is proved that non-expansivity is necessary and sufficient to guarantee equi-continuity of the values. As a consequence, pointwise and uniform convergence of $\{v_{\lambda}\}$ are equivalent, and since $X\times Y$ is compact, to prove this convergence it is enough to prove uniqueness of a limit point555By convergence or limit point of $\{v_{\lambda}\}$, we always mean when $\lambda$ approaches 0..

Remark 2.5**.**

It is not difficult to see that without non-expansivity, $\{v_{\lambda}\}$ may not be equicontinuous and the convergence may not be uniform. For instance in red-and-black casino with a single player, if the parameter $w>\frac{1}{2}$ and $u(x)=x$, $v_{\lambda}$ is continuous for every $\lambda$ but $v={\rm lim}_{{}_{\lambda}\rightarrow 0}v_{\lambda}$ is not : $v(x)=0$ for $x=0$ and $v(x)=1$ for $x>0$.

2.5 Excessive, depressive and balanced functions

Definition 2.6**.**

• Let $v$ be in ${\cal B}(X\times Y)$.*

1) $v$ is balanced if $\forall(x,y)\in X\times Y$,

$v(x,y)=\max_{p\in\Gamma(x)}\min_{q\in\Gamma(y)}\tilde{v}(p,q)=\min_{q\in\Gamma(y)}\max_{p\in\Gamma(x)}\tilde{v}(p,q).$

2) $v$ is excessive (with respect to $X$) if: $\forall(x,y)\in X\times Y$, $v(x,y)=\max_{p\in\Gamma(x)}\tilde{v}(p,y).$

3) $v$ is depressive (with respect to $Y$) if: $\forall(x,y)\in X\times Y$, $v(x,y)=\min_{q\in\Gamma(y)}\tilde{v}(x,q).$

Observe that any uniform limit $v$ of $(v_{\lambda})_{\lambda\in(0,1]}$ is necessarily continuous and balanced (by passing to the limit in definition 2.1). In a splitting game, excessive means concave with respect to the first variable, and depressive means convex with respect to the second variable.

Example 2.7**.**

Consider a gambling game where players 1 and 2 move on the same finite grid of a circle containing $6$ nodes in equidistant positions. Any player can move one step to the left, one step to the right, or not move (and choose randomly between these 3 options, so that transitions have convex values). It is here possible for a player to go from any state to any other state in at most $3$ stages (the game may be called cyclic), so any excessive and depressive function is necessarily constant. Suppose that Player 1’s payoff is 1 if he is at most one step away from Player 2, and Player 1’s payoff is 0 otherwise. If the players start a distance at most 1, Player 1 can guarantee this property will hold forever by not moving or moving one step to the direction of Player 2 and so, in this case we have $v_{\lambda}=1$ for every $\lambda$. On the other hand, if the players start at a distance at least $2$, Player 2 can insure that this property will hold forever, by not moving or moving one step in the opposite direction of Player 1. For these initial states, $v_{\lambda}=0$ for every $\lambda$. Here, $v={\rm lim}\,v_{\lambda}$ is continuous and balanced, but not excessive nor depressive.

2.6 Acyclicity

We now come to the main definitions of the paper.

Definition 2.8**.**

**

1) The gambling game is leavable if: $\forall x\in X,\delta_{x}\in\Gamma(x)\;{\it and}\;\forall y\in Y,\delta_{y}\in\Lambda(y).$

2) The gambling house $\Gamma$ of player 1 is weakly acyclic if there exists $\varphi$ in ${\cal B}(X)$ lower semi-continuous such that:

[TABLE]

Similarly, the gambling house $\Lambda$ of player 2 is weakly acyclic if there exists $\psi$ in ${\cal B}(Y)$ upper semi-continuous such that:

[TABLE]

The gambling game is weakly acyclic if both gambling houses are weakly acyclic.

The gambling game is leavable if each player can remain in any given state. This is a standard assumption [6]. This is the case in red-and-black casinos and splitting games. In persuasion games and in repeated games with incomplete information, not moving means revealing no information. If the game is leavable, any excessive and depressive function is necessarily balanced. The converse is not necessarily true as example 2.7 shows.

Weak acyclicity is, to our knowledge, a new condition in the gambling house literature. If the house $\Gamma$ is weakly acyclic, the “potential” $\varphi$ decreases in expectation along non stationary trajectories, hence the irreversibility of the process (in the space of probabilities over $X$).

Observe that any weakly acyclic gambling game is necessarily leavable. When $w\leq\frac{1}{2}$, a red-and-black casino is weakly acyclic (take $\varphi$ to be strictly increasing and strictly concave). Also, a splitting game is weakly acyclic (take $\varphi$ to be any strictly concave function on $X$).

We will now define strong acyclicity, our central condition. For this, we need to consider transitions for several stages. We first extend linearly the transitions to $\Delta(X)$ and $\Delta(Y)$ by defining $\tilde{\Gamma}:\Delta(X)\rightrightarrows\Delta(X)$ and $\tilde{\Lambda}:\Delta(Y)\rightrightarrows\Delta(Y)$. More precisely, the graph of $\tilde{\Gamma}$ is defined as the closure of the convex hull of the graph of $\Gamma$ (viewed as the subset $\{(\delta_{x},p),x\in X,p\in\Gamma(x)\}$ of $\Delta(X)\times\Delta(X)$), and similarly the graph of $\tilde{\Lambda}$ is defined as the closed convex hull of the graph of $\Lambda$. Because Dirac measures are extreme points of $\Delta(X)$ and $\Delta(Y)$, we have $\tilde{\Gamma}(\delta_{x})=\Gamma(x)$ and $\tilde{\Lambda}(\delta_{y})=\Lambda(y)$ for each $x$ in $X$ and $y$ in $Y$. Be careful that in general, for $p$ in $\Delta(X)$ and $q$ in $\Delta(Y)$: $\tilde{v}_{\lambda}(p,q)\neq\max_{p^{\prime}\in\tilde{\Gamma}(p)}\min_{q^{\prime}\in\tilde{\Lambda}(q)}\left(\lambda u(p,q)+(1-\lambda){\tilde{v}}_{\lambda}(p^{\prime},q^{\prime})\right).$

We now define inductively a sequence of transitions $(\tilde{\Gamma}^{n})_{n}$ from $\Delta(X)$ to $\Delta(X)$, by $\tilde{\Gamma}^{0}(p)=\{p\}$ for every state $p$ in $\Delta(X)$, and666The composition being defined by $G\circ H(p)=\{p"\in\Delta(X),\exists p^{\prime}\in H(p)\;{\rm s.t.}\;p"\in G(p^{\prime})\}$. for each $n\geq 0$, $\tilde{\Gamma}^{n+1}=\tilde{\Gamma}^{n}\circ\tilde{\Gamma}$ . $\tilde{\Gamma}^{n}(\delta_{x})$ represents the set of probabilities over states that Player 1 can reach in $n$ stages from the initial state $x$ in $X$. Similarly we define $\tilde{\Lambda}^{n}$ for each $n$.

Definition 2.9**.**

**

*1) The reachable set of Player 1 from state $x$ in $X$ is the closure of $\bigcup_{n\geq 0}\tilde{\Gamma}^{n}(\delta_{x})$ in $\Delta(X)$, and denoted $\Gamma^{\infty}(x)$. Similarly, the reachable set of Player 2 from state $y$ in $Y$ is the subset $\Lambda^{\infty}(y)$ of $\Delta(Y)$ defined as the closure of $\bigcup_{n\geq 0}\tilde{\Lambda}^{n}(\delta_{y})$.

2) The gambling house $\Gamma$ of player 1 is strongly acyclic (or simply, acyclic) if there exists $\varphi$ in ${\cal B}(X)$ lower semi-continuous such that:

[TABLE]

Similarly, the gambling house $\Lambda$ of player 2 is strongly acyclic (or simply, acyclic) if there exists $\psi$ in in ${\cal B}(Y)$ upper semi-continuous such that:

[TABLE]

The gambling game is strongly acyclic (or simply, acyclic) if both gambling houses are strongly acyclic.

Clearly, strong acyclicity implies weak acyclicity. Note that in splitting games, we have $\Gamma\circ\Gamma=\Gamma$, so $\Gamma^{\infty}=\Gamma$ and weak and strong acyclicity coincide.

3 Main Results

In all the paper (and in particular the results below), we consider standard gambling games.

Definition 3.1**.**

A gambling game is standard if both state spaces $X$ and $Y$ are compact metric, the running payoff $u$ is continuous, and the transitions $\Gamma$ and $\Lambda$ have non empty convex compact values and are leavable and non expansive.

We will also use the following properties.

Definition 3.2**.**

Given $v$ in $\mathcal{B}(X\times Y)$, we say that :

$v$* satisfies $P1$ if: $\forall(x,y)\in X\times Y,\exists p\in\Gamma^{\infty}(x),v(x,y)=\tilde{v}(p,y)\leq u(p,y),$*

$v$* satisfies $P2$ if : $\forall(x,y)\in X\times Y,\exists q\in\Lambda^{\infty}(y),v(x,y)=\tilde{v}(x,q)\geq u(x,q).$*

Our main result is the following.

Theorem 3.3**.**

Consider a standard gambling game.

*1. If at least one of the players has a strongly acyclic gambling house, $(v_{\lambda})$ uniformly converges to the unique function $v$ in $\mathcal{C}(X\times Y)$ satisfying: *

*a) $v$ is excessive , i.e. $\forall(x,y)\in X\times Y$, $v(x,y)=\max_{p\in\Gamma(x)}\tilde{v}(p,y),$

*b) $v$ is depressive, i.e. $\forall(x,y)\in X\times Y$, $v(x,y)=\min_{q\in\Gamma(y)}\tilde{v}(x,q),$

*c) $v$ satisfies $P1$, i.e. $\forall(x,y)\in X\times Y,\exists p\in\Gamma^{\infty}(x),v(x,y)=\tilde{v}(p,y)\leq u(p,y),$

*d) $v$ satisfy $P2$, i.e. $\forall(x,y)\in X\times Y,\exists q\in\Lambda^{\infty}(y),v(x,y)=\tilde{v}(x,q)\geq u(x,q).$

Moreover: $v$ is the largest excessive-depressive continuous function satisfying $P1$, and is the smallest excessive-depressive continuous function satisfying $P2$.

2. If both gambling houses are weakly acyclic, convergence of $(v_{\lambda})$ may fail.

The conditions of the positive result 1) may be interpreted as follows:

• •

a) and b) : It is always safe not to move. For each player, not moving ensures not to degrade the limit value.

• •

c) and d) : Each player can reach, if his opponent does not move, the zone when the current payoff is at least as good than the limit value, without degrading the limit value.

These interpretations will lead later to the construction of simple uniformly optimal strategies in some gambling games, see section 7.3.

The positive result of theorem 3.3 relies on the following three propositions. Recall that thanks to proposition 2.4, to get convergence of the values it is enough to show uniqueness of a limit point of $(v_{\lambda})_{\lambda}$.

Proposition 3.4**.**

Assume one of the player has a weakly acyclic gambling house. If $v$ in $\mathcal{C}(X\times Y)$ is balanced, then $v$ is excessive and depressive.

Proposition 3.5**.**

Let $v$ be a limit point of $(v_{\lambda})$ for the uniform convergence. Then $v$ is balanced, and satisfies $P1$ and $P2$.

Proposition 3.6**.**

Assume one of the player has a strongly acyclic gambling house. Then, any balanced continuous function satisfying $P1$ is smaller that any balanced continuous function satisfying $P2$. Consequently, there is at most one balanced continuous function satisfying $P1$ and $P2$.

If none of the player has a strongly acyclic gambling house, there may be infinitely many balanced continuous functions satisfying P1 and P2. This will be the case in our counter-example of section 6, where both gambling houses are weakly acyclic.

4 Examples

4.1 A simple acyclic gambling house

Let us first illustrate our characterization on a simple example. Consider the following Markov decision process with 3 states: $X=\{a,b,c\}$ from [31]. States $b$ and $c$ are absorbing with respective payoffs 1 and 0. Start at $a$, choose $\alpha\in I=[0,1/2]$, and move to $b$ with proba $\alpha$ and to $c$ with proba $\alpha^{2}$.

a0b1c0\alpha$$1-\alpha-\alpha^{2}$$\alpha^{2}

Here formally $Y$ is a singleton (there is only one player, so we can omit the variable $y$), $\Gamma(b)=\{\delta_{b}\}$, $\Gamma(c)=\{\delta_{c}\}$, and $\Gamma(a)={\rm conv}\{(1-\alpha-\alpha^{2})\delta_{a}+\alpha\delta_{b}+\alpha^{2}\delta_{c},\alpha\in I\}.$ The payoffs are $u(a)=u(c)=0$, $u(b)=1$.

$\Gamma$ has compact convex values, the transitions are 1-Lipschitz, and the game is leavable. The gambling game is strongly acyclic: just consider $\varphi$ such that $\varphi(a)=1$, and $\varphi(b)=\varphi(c)=0$.

Player 1 can go from state $a$ to state $b$ in infinitely many stages with arbitrarily high probability, by repeating a choice of $\alpha>0$ small (so that $\alpha^{2}$ is much smaller than $\alpha$), and the limit value $v$ clearly satisfies:

[TABLE]

This is the unique function $w:X\rightarrow I\!\!R$ satisfying the conditions a), b), c), d) of Theorem 3.3: $P1$ and $P2$ implies $u\leq w\leq 1$, and because $b$ and $c$ are absorbing states, $w(b)=1$ and $w(c)=0$. Finally, $w$ excessive gives $w(a)=1$. Notice that $\delta_{b}\in\Gamma_{\infty}(a)$ but for each $n$, $\delta_{b}\notin\tilde{\Gamma}_{n}(a)$.

4.2 A weakly acyclic gambling house

Let us modify the gambling house of the previous section 4.1. We still have a unique player and a state space $X=\{a,b,c\}$. The only difference is that state $b$ is no longer absorbing : in state b the player also has to choose some $\alpha\in I=[0,1/2]$, and then moves to $a$ with probability $\alpha$, to $c$ with probability $\alpha^{2}$ and remains in $b$ with probability $1-\alpha-\alpha^{2}$.

abc\alpha$$1-\alpha-\alpha^{2}$$\alpha^{2}$$\alpha$$1-\alpha-\alpha^{2}$$\alpha^{2}

States $a$ and $b$ are now symmetric. This gambling house is weakly acyclic, with $\varphi(a)=\varphi(b)=1,\varphi(c)=0$, but it is not strongly acyclic since $a\in\Gamma^{\infty}(b)$ and $b\in\Gamma^{\infty}(a)$. We will later use this gambling house to construct our counter-example of theorem 3.3, 2).

4.3 An example with countable state spaces

We present here a (strongly) acyclic gambling game with countable state spaces, and illustrate777We have first studied and understood this example before proving proposition 3.4. Example 2.7 shows that weakening the assumption of weakly acyclic gambling game to leavable gambling game is not possible in this proposition. the proof of proposition 3.4, that under weak acyclicity any continuous balanced function is also excessive and depressive. Consider the state space:

[TABLE]

where $x_{n}=1-\frac{1}{n}$ if $n$ is finite, and $x_{\infty}=1$. We use $d(x,x^{\prime})=|x-x^{\prime}|$, so that $X$ is countable and compact. The transition is given by:

[TABLE]

The intuition is clear: Player 1 can stay at his location, or move 1 to the right. The gambling house $(Y,\Lambda)$ of Player 2 is a copy of the gambling house of Player 1. Transitions are non expansive (since $|\frac{1}{n+1}-\frac{1}{n^{\prime}+1}|\leq|\frac{1}{n}-\frac{1}{n^{\prime}}|)$, and the game is strongly acyclic. The payoff $u$ is any continuous function $X\times Y\longrightarrow I\!\!R$, so that theorem 3.3 applies.

Consider $v:X\times Y\longrightarrow I\!\!R$, and for simplicity we use $w(n,m)=v(x_{n},x_{m})$. Here $v$ excessive means that $w(n,m)$ is weakly decreasing in $n$, and $v$ depressive means that $w(n,m)$ is weakly increasing in $m$. The meaning of $v$ balanced is the following: for each $n$ and $m$, $w(n,m)$ is the value of the matrix game (“local game” at $(n,m)$):

[TABLE]

Clearly, if $v$ is excessive and depressive it is balanced, but proposition 3.4 tells that if $v$ is continuous, then the converse also holds: balancedness implies excessiveness and depressiveness. The idea of the proof of proposition 3.4 can be seen here as follows.

Suppose $v$ is balanced, but not excessive. Then one can find $n$ and $m$ such that $w({n+1},m)>w(n,m)$. Because $w(n,m)$ is the value of the local game at $(n,m)$, we necessarily have $w(n+1,m+1)\leq w(n,m)$, and $w(n+1,m)>w(n+1,m+1)$. Consider now the “local game” at $(n+1,m)$. $w(n+1,m)$ is the value of the matrix:

[TABLE]

Since $w(n+1,m)>w(n+1,m+1)$, we obtain $w(n+2,m+1)\geq w(n+1,m).$ We have obtained: $w(n+2,m+1)\geq w(n+1,m)>w(n,m)\geq w(n+1,m+1)$, so

[TABLE]

Iterating the argument, we obtain that for each $p$,

[TABLE]

And this is a contradiction with $w$ being continuous at infinity.

To conclude with this example, consider the simple case where the running payoff is given by $u(x,y)=|x-y|$. Player 1 wants to be far from Player 2, and Player 2 wants to be close to Player 1. If initially $n<m$, it is optimal for each player not to move, so $w(n,m)=|x_{n}-x_{m}|$. Suppose on the contrary that initially $n\geq m$, so that Player 1 is more to the right than Player 2. Then Player 2 has a simple optimal strategy which is to move to the right if the current positions satisfy $x>y$, and to stay at $y$ if $x=y$. No matter how large is the initial difference $n-m$, Player 2 will succeed in being close to player, so that $w(n,m)=0$ if $n\geq m$.

5 Proof of Theorem 3.3, part 1.

We prove here propositions 3.4, 3.5 and 3.6.

5.1 Proof of proposition 3.4

By symmetry, suppose that $\Gamma$ is weakly acyclic and $\Lambda$ Leavable. Let us prove that any balanced continuous function $v$ is excessive-depressive. First let us first prove that $v$ is excessive.

Fix any $(x_{0},y_{0})$ in $X\times Y$, and $p_{1}\in\Gamma(x_{0})$. A direct consequence of balancedness is the existence of $q_{1}$ in $\Lambda(y_{0})$ such that $v(p_{1},q_{1})\leq v(x_{0},y_{0})$. Now, $p_{1}$ is in $\Delta(X)$ and $q_{1}$ is in $\Delta(Y)$. One has to be careful that there may not exist $p_{2}\in\tilde{\Gamma}(p_{1})$ such that for all $q_{2}\in\tilde{\Lambda}(q_{1})$, $v(p_{2},q_{2})\geq v(p_{1},q_{1})$. This is because, $\tilde{v}$ being affine in each variable, $v(p_{1},q_{1})$ can be interpreted as the value of the auxiliary game where first $x$ and $y$ are chosen according to $p_{1}\otimes q_{1}$ and observed by the players, then players respectively choose $p\in\Gamma(x)$ and $q\in\Lambda(y)$ and finally Player 1’s payoff is $v(p,q)$. And to play well in this game Player 1 has to know the realization of $q_{1}$ before choosing $p$. However since $y_{0}$ is a Dirac measure, balancedness implies that there exists $p_{2}\in\tilde{\Gamma}(p_{1})$ such that $v(p_{2},q_{1})\geq v(p_{1},y_{0}).$ We have obtained the following lemma:

Lemma 5.1**.**

Given $(x_{0},y_{0})$ in $X\times Y$, and $p_{1}\in\Gamma(x_{0})$, there exists $q_{1}$ in $\Lambda(y_{0})$ and $p_{2}\in\tilde{\Gamma}(p_{1})$ such that: $v(p_{1},q_{1})\leq v(x_{0},y_{0})\;{\rm and}\;v(p_{2},q_{1})\geq v(p_{1},y_{0}).$

We now prove the proposition. Define, for $x$ in $X$,

[TABLE]

$h$ is continuous. We put $Z={\rm Argmax}_{x\in X}h(x)$, and consider $x_{0}\in{\rm Argmin}_{x\in Z}\varphi(x)$, where $\varphi$ comes from the definition of $\Gamma$ weakly acyclic.

$x_{0}\in Z$, so $v$ is excessive if and only if $h_{0}=_{def}h(x_{0})\leq 0$. By definition of $h(x_{0})$, there exists $p_{1}\in\Gamma(x_{0})$ and $y_{0}$ in $Y$ such that $v(p_{1},y_{0})-v(x_{0},y_{0})=h_{0}.$

By lemma 5.1, there exists $q_{1}$ in $\Lambda(y_{0})$ and $p_{2}\in\tilde{\Gamma}(p_{1})$ such that $v(p_{1},q_{1})\leq v(x_{0},y_{0})$ and $v(p_{2},q_{1})\geq v(p_{1},y_{0}).$ Consequently,

[TABLE]

One can now find $y_{1}$ in $Y$ such that $v(p_{2},y_{1})-v(p_{1},y_{1})\geq h_{0}$, and since $p_{2}\in\tilde{\Gamma}(p_{1})$ it implies that ${\rm Supp}(p_{1})\subset Z$. But $p_{1}\in\Gamma(x_{0})$, so by definition of $x_{0}$ and weak acyclicity, we obtain that $p_{1}=\delta_{x_{0}}.$ So $h_{0}=0$, and $v$ is excessive.

Let us now prove that when $v$ is balanced and excessive then it is depressive. For every $(x,y)$, and every $p\in\Gamma(x)$, we have $v(x,y)\geq v(p,y)$. Thus, for every $(x,y)$, $p\in\Gamma(x)$ and every $q\in\Lambda(y)$, $v(x,q)\geq v(p,q)$ and consequently, $\min_{q\in\Lambda(y)}v(x,q)\geq\min_{q\in\Lambda(y)}v(p,q)$. Taking the maximum in $p\in\Gamma(x)$ and using that $v$ is balanced implies that $\min_{q\in\Lambda(y)}v(x,q)\geq v(x,y)$. Since $\Lambda$ is leavable, we have equality and so $v$ is depressive with respect to $Y$.

5.2 Proof of proposition 3.5

Let $(\lambda_{n})_{n}$ be a vanishing sequence of discount factors such that $\|v_{\lambda_{n}}-v\|\rightarrow_{n\to\infty}0$. Fix $(x,y)$ in $X\times Y$, by symmetry it is enough to show that there exists $p\in\Gamma^{\infty}(x)$ such that $v(x,y)\leq v(p,y)\leq u(p,y)$. If $v(x,y)\leq u(x,y)$, it is enough to consider $p=\delta_{x}$, so we assume $v(x,y)>u(x,y)$. For $n$ large enough, $v_{\lambda_{n}}(x,y)>u(x,y)+\lambda_{n}$.

Fix $n$. We define inductively a sequence $(p_{t}^{n})_{t=0,...,T_{n}}$ in $\Delta(X)$, with $T_{n}\geq 1$, by:

1. $p_{0}^{n}=\delta_{x}$,

2. for each $t\geq 0$ such that $v_{\lambda_{n}}(p_{t}^{n},y)>u(p_{t}^{n},y)+\lambda_{n}$, we define $p_{t+1}^{n}$ in $\tilde{\Gamma}(p_{t}^{n})$ by:

[TABLE]

We have $\max_{p\in\tilde{\Gamma}(p_{t}^{n})}\left(\lambda_{n}u(p_{t}^{n},y)+(1-\lambda_{n})v_{\lambda_{n}}(p,y)\right)\geq v_{\lambda_{n}}(p_{t}^{n},y)$, so:

[TABLE]

Since $u(p_{t}^{n},y)<v_{\lambda_{n}}(p_{t}^{n},y)-\lambda_{n}$, we obtain:

[TABLE]

Since $\frac{\lambda^{2}_{n}}{1-\lambda_{n}}>0$ and $v_{\lambda_{n}}$ is bounded, there exists a first integer $t=T_{n}$ where $v_{\lambda_{n}}(p_{T_{n}}^{n},y)\leq u(p_{T_{n}}^{n},y)+\lambda_{n}$, and we stop here the definition of the sequence $(p_{t}^{n})_{t=0,...,T_{n}}$. Inequalities (1) and (2) above give:

[TABLE]

Define now $p^{n}=p^{n}_{T_{n}}$ for each $n$. $p^{n}\in\tilde{\Gamma}^{T_{n}}(x)\subset\Gamma^{\infty}(x)$ for each $n$, and we consider a limit point $p^{*}\in\Gamma^{\infty}(x)$ of $(p^{n})_{n}$. Because $v_{\lambda_{n}}$ is an equicontinuous family converging to $v$, we obtain the convergence (along a subsequence) of $v_{\lambda_{n}}(p_{T_{n}}^{n},y)$ to $v(p^{*},y)$. Passing to the limit in the inequality defining $T_{n}$ then gives:

[TABLE]

Finally, passing to the limit in (3) shows: $v(x,y)\leq v(p^{*},y).$

5.3 Proof of proposition 3.6

We start with a lemma.

Lemma 5.2**.**

Assume $v$ in $\mathcal{C}(X\times Y)$ is excessive. Then for all $(x_{0},y_{0})$ in $X\times Y$ and all $p\in\Gamma^{\infty}(x_{0})$, we have: $v(p,y_{0})\leq v(x_{0},y_{0}).$

Proof: $p={\rm lim}_{n}p_{n}$, with $p_{n}\in\tilde{\Gamma}^{n}(x_{0})$ for each $n$. It is enough to prove that $v(p_{n},y_{0})\leq v(x_{0},y_{0})$ for each $n$, and we do the proof by induction on $n$. The case $n=1$ is clear by definition of $v$ excessive. Since $p_{n+1}\in\tilde{\Gamma}(p_{n})$, it is enough to prove that for $p^{\prime}$ in $\Delta(X)$ and $p^{\prime\prime}\in\tilde{\Gamma}(p^{\prime})$, we have $v(p^{\prime\prime},y_{0})\leq v(p^{\prime},y_{0}).$ By definition of $\tilde{\Gamma}$, $(p^{\prime},p^{\prime\prime})$ is in the closure of ${\rm conv}(Graph\Gamma)$.

$y_{0}$ is fixed, and the function $h:p\longrightarrow v(p,y_{0})$ is affine continous on $\Delta(X)$. The set

[TABLE]

is convex and compact, and we want to show that $Graph(\tilde{\Gamma})=\overline{{\rm conv}}(Graph\Gamma)\subset D$. It it enough to prove that $Graph(\Gamma)\subset D$, and this is implied by the fact that $v$ is excessive. This concludes the proof of lemma 5.2.

We now prove the proposition. Assume one of the gambling houses is strongly acyclic, and let $v_{1}$ and $v_{2}$ satisfying the conditions of proposition 3.6 (are continuous, balanced, $v_{1}$ satisfies $P1$ and $v_{2}$ satisfies $P2$). We will show that $v_{1}\leq v_{2}$.

By symmetry, suppose that $\Gamma$ is strongly acyclic. From Proposition 3.4, $v_{1}$ and $v_{2}$ are excessive (in $X$) and depressive (in $Y$). $v_{1}-v_{2}$ being continuous on $X\times Y$, define the compact set:

[TABLE]

Consider now $\varphi$ u.s.c. given by the strong acyclicity condition of $\Gamma$. The set $Z$ being compact, there exists $(x_{0},y_{0})$ minimizing $\varphi(x)$ for $(x,y)$ in $Z$.

By $v_{2}$ satisfying $P2$, there exists $q$ in $\Lambda^{\infty}(y_{0})$ such that $v_{2}(x_{0},y_{0})=v_{2}(x_{0},q)\geq u(x_{0},q)$. Thus, there is $y^{\prime}_{0}\in{\rm Supp}(q)$ such that $v_{2}(x_{0},y^{\prime}_{0})\geq u(x_{0},y^{\prime}_{0})$.

Because $v_{1}$ is depressive, by lemma 5.2 we have $v_{1}(x_{0},q)\geq v_{1}(x_{0},y_{0})$ and we obtain $v_{1}(x_{0},q)-v_{2}(x_{0},q)\geq v_{1}(x_{0},y_{0})-v_{2}(x_{0},y_{0})$. Since $(x_{0},y_{0})$ is in $Z$, $\{x_{0}\}\times{\rm Supp}(q)\subset Z$. Thus, $(x_{0},y^{\prime}_{0})\in Z$. Obviously, $(x_{0},y^{\prime}_{0})$ also minimizes $\varphi(x)$ for $(x,y)$ in $Z$ (the minimum value remains unchanged: $\varphi(x_{0})$).

By $v_{1}$ satisfying $P1$, there exists $p$ in $\Gamma^{\infty}(x_{0})$ such that: $v_{1}(x_{0},y^{\prime}_{0})=v_{1}(p,y^{\prime}_{0})\leq u(p,y^{\prime}_{0})$. Because $v_{2}$ is excessive, by lemma 5.2 we have $v_{2}(p,y^{\prime}_{0})\leq v_{2}(x_{0},y^{\prime}_{0})$ and we obtain $v_{1}(p,y^{\prime}_{0})-v_{2}(p,y^{\prime}_{0})\geq v_{1}(x_{0},y^{\prime}_{0})-v_{2}(x_{0},y^{\prime}_{0})$. Since $(x_{0},y^{\prime}_{0})$ is in $Z$, ${\rm Supp}(p)\times\{y^{\prime}_{0}\}\subset Z$. By definition of $(x_{0},y^{\prime}_{0})$, this implies: $\varphi(p)\geq\varphi(x_{0})$. The definition of $\varphi$ now gives that $p$ is the Dirac measure on $x_{0}$. We obtain that $v_{1}(x_{0},y^{\prime}_{0})=v_{1}(p,y^{\prime}_{0})\leq u(x_{0},y^{\prime}_{0})$. So ${\rm Max}_{(x,y)\in X\times Y}v_{1}(x,y)-v_{2}(x,y)=v_{1}(x_{0},y^{\prime}_{0})-v_{2}(x_{0},y^{\prime}_{0})\leq u(x_{0},y^{\prime}_{0})-u(x_{0},y^{\prime}_{0})=0$, and thus $v_{1}\leq v_{2}$.

6 A weakly acyclic game without limit value

We conclude here the proof of theorem 3.3 by providing a counterexample to the convergence of $(v_{\lambda})$ in a weakly acyclic non expansive gambling house.

6.1 The counter-example

The states and transitions for Player 1 are as in example 4.2:

abc\alpha_{a}$$1-\alpha_{a}-\alpha_{a}^{2}$$\alpha_{a}^{2}$$\alpha_{b}$$1-\alpha_{b}-\alpha_{b}^{2}$$\alpha_{b}^{2}

The set of states of Player 1 is $X=\{a,b,c\}$. The difference with example 4.2 is that the set of possible choices for $\alpha_{a}$ and $\alpha_{b}$ may be smaller than $[0,1/2]$. Here $\alpha_{a}$ and $\alpha_{b}$ now belong to some fixed compact set $I\subset[0,1/2]$ such888Except in part 2 of theorem 6.2 where we will consider the case $I=\{0,1/4\}$. that 0 is in the closure of $I\backslash\{0\}$. Then $0\in I$, the transitions are leavable and non expansive. States $a$ and $b$ are symmetric, this gambling house is weakly acyclic, with $\varphi(a)=\varphi(b)=1,\varphi(c)=0$, but not strongly acyclic since $a\in\Gamma^{\infty}(b)$ and $b\in\Gamma^{\infty}(a)$.

The gambling house of Player 2 is a copy of the gambling house of Player 1, with state space $Y=\{a^{\prime},b^{\prime},c^{\prime}\}$ and a compact set of choices $J\subset[0,1/2]$ such that 0 is in the closure of $J\backslash\{0\}$. The unique difference between the gambling houses of the players is that $I$ and $J$ may be different. Payoffs are simple:

[TABLE]

The $u$ function can be written as follows  

$a^{\prime}$ $b^{\prime}$ $c^{\prime}$

$a$ [math] $1$ $1$

$b$ $1$ [math] $1$

$c$ $1$ $1$ [math]

 , with a clear interpretation : Player 1 and Player 2 both move on a space with 3 points, Player 2 wants to be at the same location as Player 1, and Player 1 wants the opposite.

Here the gambling game is weakly acyclic but not strongly acyclic, and the following lemma shows that the uniqueness property of proposition 3.6 fails.

Lemma 6.1**.**

Let $v:X\times Y\longrightarrow I\!\!R$. Then A) and B) are equivalent:

A) $v\;{\it is}\;{\it excessive},\;{\it depressive},$ and satisfies $P1$ and $P2$,

B) There exists $x\in[0,1]$ such that $v$ can be written:

[TABLE]

Proof: Assume $v$ satisfies $A)$. Since $u$ takes values in $[0,1]$, so does also $v$. Because $c$ and $c^{\prime}$ are absorbing, we have $v(c,c^{\prime})=u(c,c^{\prime})=0$.

Consider now $v(c,a^{\prime})$. $v$ being depressive, for any fixed $\beta^{*}>0$ in $J$ we have:

[TABLE]

and we obtain $v(c,a^{\prime})\leq\frac{1}{1+\beta^{*}}v(c,b^{\prime})$. But symmetrically we also have $v(c,b^{\prime})\leq\frac{1}{1+\beta^{*}}v(c,a^{\prime})$, and we get $v(c,a^{\prime})=v(c,b^{\prime})=0$.

Consider now $v(a,c^{\prime})$. By $P2$, we obtain that $v(a,c^{\prime})\geq u(a,c^{\prime})=1$, and $v(a,c^{\prime})=1$. Similarly, $v(b,c^{\prime})=1$.

Consider now $v(a,a^{\prime})$. $v$ being excessive, for any $\alpha>0$ in $I$ we have:

[TABLE]

Hence $v(a,a^{\prime})\geq\frac{1}{1+\alpha}v(b,a^{\prime})$, and by assumption on $I$ we obtain $v(a,a^{\prime})\geq v(b,a^{\prime})$. By symmetry of the transitions between $a$ and $b$, $v(a,a^{\prime})=v(b,a^{\prime})$. Similarly, $v(a,b^{\prime})=v(b,b^{\prime})$.

It only remains to prove that $v(a,a^{\prime})=v(a,b^{\prime})$. $v$ being depressive, for any $\beta>0$ in $J$, $v(a,a^{\prime})\leq\beta v(a,b^{\prime})+\beta^{2}v(a,c^{\prime})+(1-\beta-\beta^{2})v(a,a^{\prime}).$ By assumption on $J$, we get $v(a,a^{\prime})\leq v(a,b^{\prime})$. By symmetry of the transitions between $b$ and $b^{\prime}$, $v(a,b^{\prime})=v(a,a^{\prime})$, and $v$ satisfies $B)$.

One can easily check that $B)$ implies $A)$, and the proof of lemma 6.1 is complete.

The second part of theorem 3.3 is a direct consequence of the following result.

Theorem 6.2**.**

**

1) If $I=J=[0,1/4]$, the limit value exists and is:

[TABLE]

2) If $J=[0,1/4]$ and $I=\{0,1/4\}$, the limit value exists and is:

[TABLE]

3) If $J=[0,1/4]$ and $I=\{\frac{1}{4^{n}},n\in I\!\!N^{*}\}\cup\{0\}$, then $v_{\lambda}$ diverges.

6.2 Proof of theorem 6.2

We start with considerations valid for the 3 cases of the theorem. We fix $J=[0,1/4]$ in all the proof, and only assume for the moment that $I$ is a compact subset of $[0,1/4]$ containing [math] and $1/4$.

Consider $\lambda\in(0,1)$. It is clear that $v_{\lambda}(c,c^{\prime})=0$, and $v_{\lambda}(a,c^{\prime})=v_{\lambda}(b,c^{\prime})=1$. By symmetry of the payoffs and transitions, we have $v_{\lambda}(a,a^{\prime})=v_{\lambda}(b,b^{\prime})$, $v_{\lambda}(b,a^{\prime})=v_{\lambda}(a,b^{\prime})$ and $v_{\lambda}(c,c^{\prime})=v_{\lambda}(c,b^{\prime})$, so we can write $v_{\lambda}$ as:

[TABLE]

with $x_{\lambda}$, $y_{\lambda}$ and $z_{\lambda}$ in $(0,1)$.

$z_{\lambda}$ is indeed easy to compute. If the game is at $(c,a^{\prime})$, Player 1 can not move, and Player 2 wants to reach $c^{\prime}$ as fast as possible, so he will choose $\beta=1/4$ and we have (see definition 2.8): $z_{\lambda}=\lambda 1+(1-\lambda)(\frac{1}{16}0+\frac{15}{16}z_{\lambda}),$ so that:

[TABLE]

Proposition 6.3**.**

Assume $J=[0,1/4]$, $\min I=0$ and $\max I=1/4$. Then for $\lambda$ small enough,

[TABLE]

[TABLE]

[TABLE]

(6) express the fact that at $(a,a^{\prime})$ or $(b,b^{\prime})$, it is optimal for Player 2 to play the pure strategy $\beta=0$ (stay at the same location and wait until Player 1 has moved), and Player 1 can play a pure strategy $\alpha$ there. Similarly, (7) express the fact that at $(a,b^{\prime})$ or $(b,a^{\prime})$, it is optimal for Player 1 not to move. In spite of these simple intuitions, the proof of the proposition is rather technical, and relegated to the appendix.

Taking for granted proposition 6.3, we now proceed to the proof of theorem 6.2. It is simple to study the simple maximization problem of Player 2 given by (7), which is simply minimizing a concave polynomial on the interval $J=[0,1/4]$.

If $y_{\lambda}-x_{\lambda}\leq\frac{1}{2}(1-y_{\lambda})$, the minimum in (7) is achieved for $\beta=\frac{y_{\lambda}-x_{\lambda}}{2(1-y_{\lambda})}$, and otherwise it is achieved for $\beta=\frac{1}{4}$. Hence for $\lambda$ small enough:

[TABLE]

[TABLE]

Notice that the 2 inequalities of (9) imply $1+15\lambda\geq 2(1-\lambda)$, which is is not possible for $\lambda$ small. Consequently, for small discount factors the right hand side of (8) holds, and we have proved, for $\lambda$ small enough, the main equality of the proof:

[TABLE]

This clearly implies:

[TABLE]

For each $\lambda>0$, denote by $\alpha_{\lambda}\in I$ a maximizer in the expression (6), so that

[TABLE]

And since $x_{\lambda}>0$, $\alpha_{\lambda}>0$.

The following lemma implies part 2) of theorem 6.2.

Lemma 6.4**.**

If 0 is an isolated point in $I$, then $y_{\lambda}$ and $x_{\lambda}$ converge to 0.

Proof: In this case there exists $\alpha^{*}>0$ such that $\alpha_{\lambda}\geq\alpha^{*}$ for all $\lambda$. Passing to the limit in (11) gives the result.

We will now prove parts 1) and 3) of the theorem. The fact that $I\subset J$ gives an advantage to Player 2, which can be quantified as follows.

Lemma 6.5**.**

Assume that $y_{\lambda_{n}}$ and $x_{\lambda_{n}}$ converge to $v$ in $[0,1]$. Then $v\leq 1/2,$ and $y_{\lambda_{n}}-x_{\lambda_{n}}\sim 2\sqrt{\lambda_{n}}(1-v).$

Proof: $(1-\lambda)(y_{\lambda}-x_{\lambda})=2\sqrt{\lambda}\sqrt{1-\lambda}(1-y_{\lambda})$, so

[TABLE]

since $\lambda+\alpha_{\lambda}^{2}-2\alpha_{\lambda}\sqrt{\lambda}\geq 0$. Dividing par $\alpha_{\lambda}\sqrt{\lambda}$ and passing to the limit gives $2(1-v)\geq 2v$, so $v\leq 1/2,$ and the lemma is proved.

Consider again the concave optimization problem of Player 1 given by equation (6), and denote by $\alpha^{*}(\lambda)=\frac{y_{\lambda}-x_{\lambda}}{2(x_{\lambda}-z_{\lambda})}>0$ the argmax of the unconstrained problem if Player 1 could choose any $\alpha\geq 0$. If $y_{\lambda}$ and $x_{\lambda}$ converge to $v>0$, then $\alpha^{*}(\lambda)\sim\sqrt{\lambda}\frac{1-v}{v}$, and Player 1 would like to play in the $\lambda$-discounted game at $(a,a^{\prime})$ some $\alpha$ close to $\sqrt{\lambda}\frac{1-v}{v}$.

Lemma 6.6**.**

Let $\lambda_{n}$ be a vanishing sequence of discount factors such that $\sqrt{\lambda_{n}}\in I$ for each $n$. Then $y_{\lambda_{n}}$ and $x_{\lambda_{n}}$ converge to 1/2.

Proof: By considering a converging subsequence we can assume that $y_{\lambda_{n}}$ and $x_{\lambda_{n}}$ converge to some $v$ in $[0,1]$. By the previous lemma, $v\leq 1/2$, and we have to show that $v\geq 1/2$. We have for each $\lambda$ in the subsequence, since Player 1 can choose to play $\alpha=\sqrt{\lambda}$:

[TABLE]

so

[TABLE]

By passing to the limit, we get $2v\geq 2(1-v)$, and $v\geq 1/2$.

Lemma 6.7**.**

Let $\lambda_{n}$ be a vanishing sequence of discount factors such that for each $n$, the open interval $(\frac{1}{2}\sqrt{\lambda_{n}},2\sqrt{\lambda_{n}})$ does not intersect $I$. Then $\limsup_{n}y_{\lambda_{n}}\leq 4/9$.

Proof: Suppose that (up to a subsequence) $x_{\lambda_{n}}$ and $y_{\lambda_{n}}$ converges to some $v\geq 4/9$. It is enough to show that $v=4/9$. We know that $v\leq 1/2$ by lemma 6.5, and since $\alpha^{*}(\lambda)\sim\sqrt{\lambda}\frac{1-v}{v}$ we have $\frac{1}{2}\sqrt{\lambda}\leq\alpha^{*}(\lambda)\geq 2\sqrt{\lambda}$ for $\lambda$ small in the sequence. By assumption $(\frac{1}{2}\sqrt{\lambda},2\sqrt{\lambda})$ contains no point in $I$ and the objective function of Player 1 is increasing from 0 to $\alpha^{*}(\lambda)$ and decreasing after $\alpha^{*}(\lambda)$. There are 2 possible cases:

If $\alpha_{\lambda}\leq\frac{1}{2}\sqrt{\lambda}$ we have:

[TABLE]

Dividing by $\lambda$ and passing to the limit gives: $v\leq 1-v-\frac{1}{4}v$, i.e. $v\leq\frac{4}{9}.$

Otherwise, $\alpha_{\lambda}>2\sqrt{\lambda}$ and we have:

[TABLE]

Again, dividing by $\lambda$ and passing to the limit gives: $v\leq 4(1-v)-{4}v$, i.e. $v\leq\frac{4}{9}.$

Finally, lemma 6.6 proves part 1) of theorem 6.2, whereas lemmas 6.6 and 6.7 together imply part 3), concluding the proof of theorem 6.2.

Remarks 6.8**.**

$\bullet$ In case 3) of theorem 6.2, it is possible to show, using lemma (6.7), that $\liminf x_{\lambda}=\liminf y_{\lambda}=4/9$, and $\limsup x_{\lambda}=\limsup y_{\lambda}=1/2$.

$\bullet$ It is not difficult to adapt lemma (6.7) to show the divergence of $(v_{\lambda})$ as soon as $J=[0,1/4]$ and $I$ satisfies:

a) there exists a sequence $(\lambda_{n})$ converging to 0 such that $\sqrt{\lambda_{n}}\in I$ for each $n$, and

b) there exist $\eta>0$ and a sequence $(\lambda_{n})$ converging to 0 such that for each $n$, $I$ does not intersect the interval $[\sqrt{\lambda_{n}}(1-\eta),\sqrt{\lambda_{n}}(1+\eta)]$.

$\bullet$ It is important for the counterexample that $I=\{\frac{1}{2^{2n}},n\in I\!\!N^{*}\}\cup\{0\}$ is not semi-algebraic. Indeed, it has been showed that if we assume $X$ and $Y$ finite, and the transitions $\Gamma$, $\Lambda$ and the payoff $u$ to be definable in some o-minimal structure, then $(v_{\lambda})_{\lambda}$ converges [4].

7 Corollaries and Extensions

7.1 Gambling houses (or Markov Decision Processes)

We assume here that there is a unique player, i.e. that $Y$ is a singleton. Then non expansiveness is enough to guarantee the uniform convergence of $(v_{\lambda})_{\lambda}$ (as well as the uniform value, see [26]) and the limit $v$ can be characterized as follows [27]: for all $x$ in $X$,

[TABLE]

where ${R=\{p\in\Delta(X),(p,p)\in{Graph\;\tilde{\Gamma}}\}}$ is interpreted as the set of invariant measures for the gambling house (which is not necessarily leavable here). If we moreover assume that the gambling house is leavable, then $R=\Delta(X)$ and we recover the fundamental theorem of gambling [6, 19], namely, $(v_{\lambda})$ uniformly converges to:

[TABLE]

It is also easy to see that $v(x)=\sup_{p\in\Gamma^{\infty}(x)}u(p)$ for each $x$.

Our approach will lead to other characterizations. We don’t assume any acyclicity condition in the following result.

Theorem 7.1**.**

Consider a one player (leavable and non-expansive) gambling house. Then $(v_{\lambda})$ uniformly converges to the unique function $v$ in $\mathcal{C}(X)$ satisfying: $v\;{\it is}\;{\it excessive},$ $P1:\forall x\in X,\exists p\in\Gamma^{\infty}(x),v(x)=\tilde{v}(p)\leq\tilde{u}(p),$ and $P2:v\geq u.$

Proof: From proposition 3.5 any accumulation point of $(v_{\lambda})$ is excessive and satisfies $P1$ and $P2$. Thus, we just need to show uniqueness, which is a direct consequence of the following lemma.

Lemma 7.2**.**

If $v_{1}\in\mathcal{B}(X)$ satisfies $P1$ and $v_{2}\in\mathcal{C}(X)$ is excessive and satisfies $P2$, then $v_{1}\leq v_{2}$.

Proof: Take any $x\in X$. Then there is $p\in\Gamma^{\infty}(x)$ such that $v_{1}(x)=v_{1}(p)\leq u(p)$. Because $v_{2}$ is excessive and continuous, by lemma 5.2 $v_{2}(x)\geq v_{2}(p)$. Since $v_{2}\geq u$, we have $v_{2}(p)\geq u(p)$. Consequently, one has $v_{2}(x)\geq u(p)$. Thus, $v_{1}(x)-v_{2}(x)\leq 0$, as desired.

Using the gambling fundamental theorem, we obtain new viewpoints on the characterization of the limit value in leavable gambling houses.

Corollary 7.3**.**

Consider a one player (leavable and non-expansive) gambling house. Then the asymptotic value exists and is

*(1) the smallest excessive function $v$ in $\mathcal{B}(X)$ satisfying $P2$; (2) the largest excessive function $v$ in $\mathcal{B}(X)$ satisfying $P1$; (3) the unique excessive function $v$ in $\mathcal{B}(X)$ satisfying $P1$ and $P2$. Moreover, $v$ is continuous. *

7.2 Equivalent characterizations

Definition 7.4**.**

Given $g$ in ${\cal B}(X\times Y)$, $Exc_{\Gamma}(g)$ is the smallest excessive (w.r.t. $X$) function not lower than $g$, and $Dep_{\Lambda}(g)$ is the largest depressive (w.r.t. $Y$) function not greater than $g$.

$Exc_{\Gamma}$ is usually called the réduite operator and $Dep_{\Lambda}(g)=-Exc_{\Gamma}(-g)$. In splitting games, $Exc_{\Gamma}=Cav_{X}$ is the concavification operator on $X$ and $Dep_{\Lambda}(g)=Vex_{Y}$ is the convexification operator on $Y$. We introduce the following definition by analogy with the Mertens-Zamir characterization.

Definition 7.5**.**

A function $v$ in ${\cal B}(X\times Y)$ satisfies the $MZ$-characterization if:

[TABLE]

We now introduce other properties, by analogy with the one established for splitting games and repeated games with incomplete information, see for instance [14, 15, 28].

Definition 7.6**.**

Let $v$ be in ${\cal B}(X\times Y)$.

1) For each $(x,y)$ in $X\times Y$,

$y$* is extreme for $v(x,\cdot)$ if $\arg\min_{q\in\Lambda^{\infty}(y)}\widetilde{v}(x,q)=\{\delta_{y}\}$.*

2) $v$ satisfies the $E$-characterization if:

[TABLE]

Proposition 7.7**.**

Consider a standard gambling game and let $v$ in $\mathcal{C}(X\times Y)$ be excessive and depressive. Then:

*and $v$ satisfies $MZ2$ $\implies$ $v$ satisfies $P2$ $\implies$ $v$ satisfies $E2$.

Proof. Let $v$ be a continuous excessive function that satisfies $MZ1$. Fix $y$ and define for each $x$, $f(x)=\min(v(x,y),u(x,y))$. Then, for every $x$, $v(x,y)=Exc_{\Gamma}(f)(x)$. We consider the gambling house for Player 1 where the state of Player 2 is fixed to $y$ and the payoff is given by $f$. From corollary 7.3, there is $p\in\Gamma^{\infty}(x)$ such that $v(x,y)=v(p,y)\leq f(p)$. Since $f(p)\leq u(p,y)$, $v$ satisfies $P1$.

Now, let $v$ be an excessive continuous function that satisfies $P1$. Take any $x$ and $y$ and suppose that $x$ is extreme for $v(\cdot,y)$. By $P1$, there is $p^{*}\in\Gamma^{\infty}(x)$ such that $v(x,y)=v(p^{*},y)\leq u(p^{*},y)$. Because $v$ is excessive and continuous, by lemma 5.2 we have $p^{*}\in\arg\max_{p\in\Gamma^{\infty}}\widetilde{v}(p,y)$. Because $x$ is extreme for $v(\cdot,y)$, $p^{*}=\delta_{x}$ and so, $v(x,y)\leq u(x,y)$. Consequently, $E1$ is satisfied.

By symmetry, $MZ2$ $\implies$ $P2$ $\implies$ $E2$.

Remark 7.8**.**

It is easy to find examples where $E1$ is satisfied but $MZ1$ is not. For instance, assume that $Y$ is a singleton, and that $\Gamma(x)=\Delta(X)$ for each $x$ in $X$. Consider the constant, hence excessive, functions $u=0$ and $v=1$. $v$ has no extreme points hence satisfies $E1$, but $Exc_{\Gamma}\min(u,v)=u$ and $v$ does not satisfy $MZ1$.

Proposition 7.9**.**

Consider a standard gambling game and let $v$ be an excessive-depressive function in $\mathcal{C}(X\times Y)$. Then

($\Gamma$ strongly acyclic) and ($v$ satisfies $E1$) $\implies$ ($v$ satisfies $MZ1$), and; ($\Lambda$ strongly acyclic) and ($v$ satisfies $E2$) $\implies$ ($v$ satisfies $MZ2$).

*Consequently, if the gambling game is strongly acyclic, characterizations $MZ$, $P$ and $E$ are equivalents. *

Proof. Let $v$ be excessive-depressive that satisfies $E1$. Fix $y\in Y$. We want to show that $v(x,y)=g(x,y)$ where $g=Exc_{\Gamma}(f)$ and $f=\min(u,v)$. $g$ is continuous by corollary 7.3. Since $v$ is excessive and $v\geq f$, we have $v\geq g$. Let $Z=\arg\max_{x\in X}v(x,y)-g(x,y)$ and let $x_{0}=\arg\min_{x\in Z}\varphi(x)$, where $\varphi$ comes form the definition of acyclicity. It is enough to prove that $v(x_{0},y)\leq g(x_{0},y)$.

Suppose not. We have $g(x_{0},y)\geq f(x_{0},y)$, so (1) $g(x_{0},y)\geq u(x_{0},y)$. Now, let $p_{0}\in\Gamma^{\infty}(x_{0})$ such that $v(x_{0},y)=v(p_{0},y)$. Because $g$ is excessive and continuous, $g(p_{0},y)\leq g(x_{0},y)$ (lemma 5.2). Consequently, $v(p_{0},y)-g(p_{0},y)\geq v(x_{0},y)-g(x_{0},y)$. Consequently, $p_{0}$ is supported on $Z$. Thus, $\varphi(x_{0},y)\leq\varphi(p_{0},y)$, and by strong acyclicity $p_{0}=\delta_{x_{0}}$. Thus $x_{0}$ is an extreme point of $v(\cdot,y)$. By $E_{1}$, we have (2) $v(x_{0},y)\leq u(x_{0},y)$. By (1) and (2), $v(x_{0},y)-g(x_{0},y)\leq 0$. A contradiction.

7.3 Uniform optimal strategies in idempotent games

7.3.1 Idemptotent gambling games

Definition 7.10**.**

The gambling game is idempotent if $\Gamma\circ\Gamma=\Gamma$ and $\Lambda\circ\Lambda=\Lambda$.

In that case, clearly $\Gamma=\Gamma^{\infty}$ and $\Lambda=\Lambda^{\infty}$. Any state that could be reached in several stages can be reached immediately in a single stage. This holds true for instance in splitting games. Notice also that for any $\Gamma$, the multifunction $\Gamma^{\infty}$ is idempotent. If $\Gamma\circ\Gamma=\Gamma$ , then $\Gamma(x)=\tilde{\Gamma}^{n}(\delta_{x})=\Gamma^{\infty}(x)$ for all $n$ and $x$ so if the gambling game is idempotent the notions of weak and strong acyclicity coincide.

An immediate corollary of theorem 3.3 is the following.

Corollary 7.11**.**

Consider a standard idempotent gambling game where a player has an acyclic gambling house. Then $\{v_{\lambda}\}$ converges uniformly to the unique function $v$ in $\mathcal{C}(X\times Y)$ which is excessive, depressive and satisfies:

[TABLE]

[TABLE]

7.3.2 Uniform value and optimal strategies

In repeated and stochastic games, a stronger notion of limit value is given by the uniform value. As usual, a strategy of Player 1, resp. Player 2, is a measurable rule giving at every stage $t$, as a function of past and current states, an element in $\Gamma(x_{t})$, resp. of $\Lambda(y_{t})$, where $x_{t}$ and $y_{t}$ are the states of stage $t$. A pair $(x_{1},y_{1})$ of initial states and a pair of strategies $(\sigma,\tau)$ naturally define a probability on the set of plays $(X\times Y)^{\infty}$ (with the product $\sigma$-algebra, $X$ and $Y$ being endowed with their Borel $\sigma$-algebra), which expectation is written $I\!\!E_{(x_{1},y_{1}),\sigma,\tau}$.

Definition 7.12**.**

$w\in\mathcal{B}(X\times Y)$* is the uniform value of the gambling game and both players have optimal uniform strategies if:*

There exists a strategy $\sigma$ of Player 1 that uniformly guarantees $w$: for any $\varepsilon>0$, there is $N$ such that for any any $n\geq N$ and initial states $(x_{1},y_{1})$, for any strategy $\tau$ of Player 2, $I\!\!E_{(x_{1},y_{1}),\sigma,\tau}\left(\frac{1}{n}\sum_{t=1}^{n}u(x_{t},y_{t})\right)\geq w(x_{1},y_{1})-\varepsilon$.

And similarly, there exists a strategy $\tau$ of Player 2 that uniformly guarantees $w$: for any $\varepsilon>0$, there is $N$ such that for any $n\geq N$ and initial states $(x_{1},y_{1})$, for any strategy $\sigma$ of Player 1, $I\!\!E_{(x_{1},y_{1}),\sigma,\tau}\left(\frac{1}{n}\sum_{t=1}^{n}u(x_{t},y_{t})\right)\leq w(x_{1},y_{1})+\varepsilon$.

It is known [31] that the above conditions imply similar inequalities for discounted payoffs: for Player 1, the same strategy $\sigma$ is such that for any $\varepsilon>0$, there is $\lambda_{0}>0$ satisfying: for any $\lambda\leq\lambda_{0}$, $(x_{1},y_{1})$ and $\tau$, $I\!\!E_{(x_{1},y_{1}),\sigma,\tau}\left(\lambda\sum_{t=1}^{\infty}(1-\lambda)^{t-1}u(x_{t},y_{t})\right)\geq w(x_{1},y_{1})-\varepsilon$. And if the uniform value exists, it has to be $v={\rm lim}_{\lambda}v_{\lambda}={\rm lim}_{n}v_{n}$.

7.3.3 Adapted strategies

Our main theorem 3.3 suggests particularly interesting strategies for the players. Consider again conditions $P1$ and $P2$, and fix a pair of states $(x,y)$. If $u(x,y)\geq v(x,y)$, the running payoff of Player 1 is at least as good as the payoff he should expect in the long run, so we may consider that Player 1 is “quite happy” with the current situation and in order to satisfy $P1$ it is enough for him not to move, i.e. to choose $p=\delta_{x}$. If on the contrary $u(x,y)<v(x,y)$, Player 2 is happy with the current situation and can choose $q=\delta_{y}$ to satisfy $P2$, whereas Player 1 should do something, and a possibility is to move towards a $p$ satisfying $P1$. This looks interesting for Player 1 because if Player 2 does not react, eventually the distribution on states will approach $(p,y)$ and (in expectation) Player 1 will be happy again with the current situation since $u(p,y)\geq v(p,y)$.

Definition 7.13**.**

Let $w$ in ${\cal B}(X\times Y)$.

A strategy of Player 1 is adapted to $w$ if whenever the current state is $(x,y)$, it plays $p\in\Gamma(x)$ such that $w(x,y)\leq w(p,y)\leq u(p,y)$.

A strategy of Player 2 is adapted to $w$ if whenever the current state is $(x,y)$, it plays $q\in\Gamma(y)$ such that $w(x,y)\geq w(x,q)\geq u(x,q)$.

If $w$ satisfies $Q1$, resp. $Q2$, Player 1, resp. Player 2, has a strategy adapted to $w$ (using a measurable selection theorem [1]). If moreover $w$ is excessive, we have $w(x,y)=w(p,y)\leq u(p,y)$. Mertens Zamir [23] in repeated games with incomplete information and Oliu-Barton [25] in splitting games used similar strategies derived from the $MZ$-characterization instead of the $Q$-characterization.

7.3.4 Vanishing $L_{1}$-variation

In repeated games with incomplete information or in splitting games, an important property is that any martingale on a simplex has bounded variations. This suggests the following.

Definition 7.14**.**

A gambling house $\Gamma$ has vanishing $L_{1}$-variation if for every $\varepsilon>0$, there is $N$ such that for all $n\geq N$ and any sequence $(p_{t})$ s.t. $p_{t+1}\in\widetilde{\Gamma}(p_{t})$, one has $\frac{1}{n}\sum_{t=1}^{n}d_{KR}(p_{t+1},p_{t})\leq\varepsilon$.

The proof of the next result is inspired by Oliu-Barton [25] in the framework of splitting games. He shows that Player 1 (resp. Player 2) can uniformly guarantee any excessive-depressive function satisfying $MZ1$ (resp. $MZ2$). Our proof is much shorter because it uses the new $Q$-characterization.

Proposition 7.15**.**

In a standard gambling game where $\Gamma$ has vanishing $L_{1}$-variation, if $w$ in $\mathcal{B}(X\times Y)$ is excessive-depressive and satisfies $Q1$, then a strategy of Player 1 adapted to $w$ uniformly guarantees $w$.

Proof: Fix $\varepsilon>0$. Because Lipschitz functions are dense in the set of continuous functions, there exists $K>0$ and a $K$-Lispchtiz function $u_{\varepsilon}$ that is uniformly $\varepsilon$-close to $u$. Consider $w$ in $\mathcal{B}(X\times Y)$ be an excessive-depressive function satisfying $Q1$, let $\sigma$ be a strategy of Player 1 adapted to $w$ and let $\tau$ be any strategy of Player 2. We fix the initial states and write $I\!\!E=I\!\!E_{(x_{1},y_{1}),\sigma,\tau}$.

Then, the average payoff of the $n$-stage game is:

[TABLE]

Because the gambling game is of vanishing variation, there is $N$ such that when $n\geq N$ one has $\frac{K}{n}\sum_{t=1}^{n}d_{KR}(p_{t+1},p_{t})\leq\varepsilon$. Because of $Q1$, $I\!\!E(u(x_{t+1},y_{t}))\geq I\!\!E(w(x_{t+1},y_{t}))=I\!\!E(w(x_{t},y_{t}))$. Since $w$ is depressive, $I\!\!E(w(x_{t},y_{t})\geq I\!\!E(w(x_{t},y_{t-1}))$, so that $I\!\!E(w(x_{t+1},y_{t}))\geq I\!\!E(w(x_{t},y_{t-1}))$, and this property holds for every $t$. Consequently, $I\!\!E(u(x_{t+1},y_{t}))\geq I\!\!E(w(x_{2},y_{1}))$ $=w(x_{1},y_{1})$. We obtain finally $\frac{1}{n}I\!\!E(\sum_{t=1}^{n}u(x_{t},y_{t}))\geq w(x_{1},y_{1})-3\varepsilon$, ending the proof.

Corollary 7.16**.**

If $\Gamma$ and $\Lambda$ have vanishing $L_{1}$-variations then there is at most one excessive-depressive function in $\mathcal{B}(X\times Y)$ satisfying $Q1$ and $Q2$.

Proof: If such a function exists, it can be guaranteed by both players. So it must be the uniform value of the game, which is unique whenever it exists.

Combining proposition 3.5, proposition 7.15 and corollary 7.16, we obtain the existence of the uniform value in a class of gambling games:

Theorem 7.17**.**

In a standard and idempotent gambling game where $\Gamma$ and $\Lambda$ have vanishing $L_{1}$-variation, the uniform value $v$ exists, and strategies adapted to $v$ are uniformly optimal. Moreover, $v$ is the unique excessive-depressive function in $\mathcal{B}(X\times Y)$ satisfying $Q1$ and $Q2$.

Proof: From proposition 3.5, any accumulation point of $v_{\lambda}$ satisfies $P1$ and $P2$, i.e. $Q1$ and $Q2$, so is unique and is the uniform value, as shown above.

Observe that acyclicity is not assumed in theorem 7.17. But vanishing $L_{1}$-variation is a form of acyclicity (it rules out for example non-constant periodic orbits). A more formal link between acyclicity and vanishing variation is given in section 9.3 of the Appendix.

8 Open problems and future directions

We introduce the class of gambling games. It is a sub-class of stochastic games which includes MDP problems, splitting games and product stochastic games. We define a strong notion of acyclicity under which we prove existence of the asymptotic value $v$ and we establish several characterizations of $v$ which are linked to the Mertens-Zamir system of functional equations (re-formulated in our more general set-up). We also prove that our condition is tight: a slight weakening of acyclicity implies non-existence of the asymptotic value. Our example is the first in the class of product stochastic games and is probably the simplest known counterexample of convergence for finite state spaces and compact action set (the first counterexample in this class was established by Vigeral [33]). Many questions merit to be investigated in a future research:

• •

In standard gambling games, is it possible to characterize the asymptotic value in models where we know it exists (for example when $X$ and $Y$ are finite, transition function is polynomial and $\Gamma$ and $\Lambda$ are definable [4] in an o-minimal structure)? We know that the asymptotic value is excessive-depressive and satisfy P1 and P2, but this does not fully characterize it.

• •

Is there an asymptotic value if one house is strongly acyclic and the other not necessarily leavable? As seen, even when both houses are weakly acyclic, we may have divergence: strong acyclicity of one of the two houses is necessary.

• •

Is there a strongly acyclic gambling game where the uniform value fails to exist?

• •

Recently splitting games have been extended to continuous time and linked to differential games with incomplete information [10]. How our model and results extend to continuous time? The one player game has been investigated [5, 18].

9 Appendix

9.1 Proof of proposition 2.4

$u$ being uniformly continuous over the compact set $X\times Y$, we consider a concave modulus of continuity $\omega:I\!\!R_{+}\longrightarrow I\!\!R_{+}$:

[TABLE]

$\omega$ is non decreasing, concave and ${\rm lim}_{0}\;\omega=0$. Denote by ${\cal C}$ the set of functions $v$ in $\mathcal{C}(X\times Y)$ satisfying: $|v(x,y)-v(x^{\prime},y^{\prime})|\leq\omega(d(x,x^{\prime})+d(y,y^{\prime})),\;\forall x,x^{\prime}\in X,\forall y,y^{\prime}\in Y.$ We start with a lemma.

Lemma 9.1**.**

For $v$ in ${\cal C}$, $p$, $p^{\prime}$ in $\Delta(X)$, $q$, $q^{\prime}$ in $\Delta(Y)$,

[TABLE]

Proof of lemma 9.1: By the Kantorovich duality theorem, there exists $\mu$ in $\Delta(X\times X)$ with first marginal $p$ and second marginal $p^{\prime}$ satisfying: $d_{KR}(p,p^{\prime})=\int_{X\times X}d(x,x^{\prime})d\mu(x,x^{\prime}).$ Similarly there exists $\nu$ in $\Delta(Y\times Y)$ with first marginal $q$ and second marginal $q^{\prime}$ satisfying: $d_{KR}(q,q^{\prime})=\int_{Y\times Y}d(y,y^{\prime})d\nu(y,y^{\prime}).$ We have for all $x$, $x^{\prime}$, $y$, $y^{\prime}$:

[TABLE]

We integrate the above inequality with respect to the probability $\mu\otimes\nu$, and obtain using the concavity of $\omega$:

[TABLE]

We now return to the proof of proposition 2.4. Fix $\lambda$ in $(0,1]$. Given $v$ in ${\cal C}$, define $\Phi(v):X\times Y\longrightarrow I\!\!R$ by: $\Phi(v)(x,y)=\sup_{p\in\Gamma(x)}\inf_{q\in\Lambda(y)}\lambda\;u(x,y)+(1-\lambda)\;\tilde{v}(p,q).$

Consider the zero-sum game with strategy spaces $\Gamma(x)$ and $\Lambda(y)$ and payoff function $(p,q)\mapsto\lambda\;u(x,y)+(1-\lambda)\;\tilde{v}(p,q)$. The strategy spaces are convex compact and the payoff function is is continuous and affine in each variable, hence by Sion’s theorem we have:

[TABLE]

Consider $(x,y)$ and $(x^{\prime},y^{\prime})$ in $X\times Y$, and let $p$ in $\Gamma(x)$ be an optimal strategy of Player 1 in the zero-sum game corresponding to $(x,y)$. The gambling game has non expansive transitions, so there exists $p^{\prime}\in\Gamma(x^{\prime})$ such that $d_{KR}(p,p^{\prime})\leq d(x,x^{\prime})$. Consider any $q^{\prime}$ in $\Lambda(y^{\prime})$, there exists $q$ in $\Gamma(y)$ with $d_{KR}(q,q^{\prime})\leq d(y,y^{\prime})$. Now, using lemma 9.1 we write:

$\lambda u(x^{\prime},y^{\prime})+(1-\lambda)\tilde{v}(p^{\prime},q^{\prime})$

$\geq\lambda(u(x,y)-\omega(d(x,x^{\prime})+d(y,y^{\prime}))+(1-\lambda)(\tilde{v}(p,q)-\omega({d_{KR}(p,p^{\prime})+d_{KR}(q,q^{\prime})})),$

$\geq\Phi(v)(x,y)-\omega({d(x,x^{\prime})+d(y,y^{\prime})}).$

We obtain $\Phi(v)(x^{\prime},y^{\prime})\geq\Phi(v)(x,y)-\omega({d(x,x^{\prime})+d(y,y^{\prime}))}$, and $\Phi(v)$ belongs to ${\cal C}$.

The rest of the proof is very standard. ${\cal C}$ is a complete metric space for $\|v-w\|=\sup_{(x,y)\in X\times Y}|v(x,y)-w(x,y)|$, and we have $\|\Phi(v)-\Phi(w)\|\leq(1-\lambda)\|v-w\|$, so $\Phi$ is $(1-\lambda)$-contracting. Hence $\Phi$ has a unique fixed point which is $v_{\lambda}$. Each $v_{\lambda}$ is in ${\cal C}$, and we obtain that the family $(v_{\lambda})_{\lambda\in(0,1]}$ is equicontinuous, ending the proof of proposition 2.4.

9.2 Proof of proposition 6.3

We proceed in 4 steps.

1. The game at $(b,a^{\prime})$: It is intuitively clear that $y_{\lambda}\geq x_{\lambda}$ since Player 1 is better off when the players have different locations. We now formalize this idea. Consider the game at $(b,a^{\prime})$. The current payoff is 1, and Player 1 has the option not to move, so we obtain by definition 2.8:

[TABLE]

This implies

[TABLE]

and since $y_{\lambda}\leq 1$, we obtain: $x_{\lambda}\leq y_{\lambda}$. Now, $\min_{\beta\in J}(\beta(x_{\lambda}-y_{\lambda})+\beta^{2}(1-y_{\lambda}))\geq\min_{\beta\in J}\beta(x_{\lambda}-y_{\lambda})+\min_{\beta\in J}\beta^{2}(1-y_{\lambda})=1/4(x_{\lambda}-y_{\lambda})$, hence:

[TABLE]

In the same spirit, in the game at $(a,a^{\prime})$, Player 2 has the option not to move, so we have:

[TABLE]

Hence,

[TABLE]

2. The game at $(a,a^{\prime})$: Consider now the game at $(a,a^{\prime})$. By definition 2.8, $x_{\lambda}$ is the value of the game (possibly played with mixed strategies), where Player 1 chooses $\alpha$ in $I$, Player 2 chooses $\beta$ in $J$ and the payoff to Player 1 is: $(1-\lambda)g_{\lambda}(\alpha,\beta)$, where $g_{\lambda}(\alpha,\beta)=$

[TABLE]

We want to prove that in this game, it is a dominant strategy for Player 2 not to move, that is to choose $\beta=0$. We need to show that for all $\alpha$ and $\beta$, $g_{\lambda}(\alpha,0)\leq g_{\lambda}(\alpha,\beta)$. As a function of $\beta$, $g_{\lambda}(\alpha,\beta)$ can be written as a constant plus:

[TABLE]

So we want to show that for all $\alpha$ in $I$, $\beta$ in $J$:

[TABLE]

Since the expression is decreasing in $\alpha$, it is enough to prove it with $\alpha=1/4$:

[TABLE]

This is true for $\beta=0$, and will be true for all $\beta$ in $J$ if and only if it is true for $\beta=1/4$, so we are left with proving:

[TABLE]

Consider $\lambda\leq 1/32$, and recall that $z_{\lambda}\leq 16\lambda$. If $x_{\lambda}\leq 1/2$, then clearly (14) holds. Assume on the contrary that $x_{\lambda}\geq 1/2$, then $z_{\lambda}\leq x_{\lambda}$, and (13) gives: $\lambda x_{\lambda}\leq(1-\lambda)\frac{1}{4}(y_{\lambda}-x_{\lambda})+0$, so $(y_{\lambda}-x_{\lambda})\geq 2\lambda$ and (14) holds as well.

We have shown that in the $\lambda$-discounted game at $(a,a^{\prime})$ with $\lambda\leq 1/32$, Player 2 has a pure dominant strategy which is $\beta=0$. Considering a pure best reply of Player 1 against this strategy implies that the game at $(a,a^{\prime})$ has a value in pure strategies satisfying $x_{\lambda}=(1-\lambda)\max_{\alpha\in I}(\alpha y_{\lambda}+(1-\alpha-\alpha^{2})x_{\lambda}+\alpha^{2}z_{\lambda})$, i.e. equation (6) is proved.

3. Small discount factors: For the sake of contradiction, assume that $z_{\lambda_{n}}\geq x_{\lambda_{n}}$ for a vanishing sequence $\lambda_{n}$ of discount factors. Then by equation (6), we have for each $n$:

[TABLE]

Since $z_{\lambda_{n}}$ converges to 0, so does $x_{\lambda_{n}}$ and $y_{\lambda_{n}}$, and moreover $\frac{y_{\lambda_{n}}-x_{\lambda_{n}}}{\lambda_{n}}$ converges to 0. This is in contradiction with equation (12). We have shown (5).

4. The game at $(b,a^{\prime})$ again: We proceed as for the game at $(a,a^{\prime})$ and will show that in the game at $(b,a^{\prime})$, it is a dominant strategy for Player 1 not to move. By definition, $y_{\lambda}$ is the value of the game where Player 1 chooses $\alpha$ in $I$, Player 2 chooses $\beta$ in $J$ and the payoff is $\lambda+(1-\lambda)h_{\lambda}(\alpha,\beta)$, with $h_{\lambda}(\alpha,\beta)=$

[TABLE]

We want to show that $h_{\lambda}(0,\beta)\geq h_{\lambda}(\alpha,\beta)$ for all $\alpha$ and $\beta$. That is, for all $\alpha$ and $\beta$,

[TABLE]

For $\lambda$ small enough, we have $z_{\lambda}\leq x_{\lambda}\leq y_{\lambda}$, and the above property is satisfied. Hence in the game at $(b,a^{\prime})$ it is dominant for Player 1 to choose $\alpha=0$. Consequently, Player 2 has a pure optimal strategy and we can write:

[TABLE]

proving equation (7). And the proof of proposition 6.3 is complete.

9.3 $L_{2}$-variation, $L_{1}$-variation and acyclicity

Definition 9.2**.**

A gambling house $\Gamma$ is of bounded $L_{2}$-variation if there is $C>0$ such that for every sequences $\{p_{t}\}$ satisfying $p_{t+1}\in\widetilde{\Gamma}(p_{t})$, one has

[TABLE]

For example splitting games have bounded $L_{2}$-variation.

Proposition 9.3**.**

If $\Gamma$ is idempotent, non-expansive, leavable and of bounded $L_{2}$-variation, then it is acyclic and has vanishing $L_{1}$-variation.

Proof: Bounded $L_{2}$-variation $\implies$ weak acyclicity because the real valued function:

[TABLE]

is strictly decreasing along non-constant orbits of $\widetilde{\Gamma}$ (i.e. $\arg\max_{p\in\Gamma(x)}\varphi(p)=\delta_{x}$). But for idempotent $\Gamma$, weak acyclicity and acyclicity coincide because $\Gamma^{\infty}=\Gamma$. Continuity of $\varphi$ is a consequence of non-expansivity of $\Gamma$ and the fact the bound $C$ is uniform over the sequences $\{p_{t}\}$ satisfying $p_{t+1}\in\widetilde{\Gamma}(p_{t})$. Finally, that bounded $L_{2}$-variation implies vanishing $L_{1}$-variation is a consequence of Cauchy-Schwartz inequality ($\frac{1}{n}\sum_{t=1}^{n}d_{KL}(p_{t+1},p_{t})\leq\frac{1}{\sqrt{n}}\sqrt{\sum_{t=1}^{n}d_{KL}(p_{t+1},p_{t})^{2}}$).