Toward Solving 2-TBSG Efficiently

TL;DR

This paper introduces strongly polynomial algorithms for solving 2-TBSG, a two-player game model, by developing novel simplex strategy iteration methods inspired by Markov decision processes, and transforms general instances into simplified forms.

Contribution

It presents the first strongly polynomial algorithms for 2-TBSG and introduces a transformation method to reduce general cases to simpler two-action scenarios.

Findings

Algorithms exhibit geometric convergence.

Proved strongly polynomial complexity.

Transformation reduces problem complexity.

Abstract

2-TBSG is a two-player game model which aims to find Nash equilibriums and is widely utilized in reinforced learning and AI. Inspired by the fact that the simplex method for solving the deterministic discounted Markov decision processes (MDPs) is strongly polynomial independent of the discounted factor, we are trying to answer an open problem whether there is a similar algorithm for 2-TBSG. We develop a simplex strategy iteration where one player updates its strategy with a simplex step while the other player finds an optimal counterstrategy in turn, and a modified simplex strategy iteration. Both of them belong to a class of geometrically converging algorithms. We establish the strongly polynomial property of these algorithms by considering a strategy combined from the current strategy and the equilibrium strategy. Moreover, we present a method to transform general 2-TBSGs into special 2-TBSGs where each state has exactly two actions.

Equations

$$v^{{\pi }_1,{\pi }_2}=v^{\pi }=(I-\gamma P_{\pi }{)}^{-1}{r}_{\pi }.$$

$$r^{\pi }=r-(J-\gamma P)v_{\pi },$$

$$J_{ji}=\{\begin{array}{ll}1& \text{if }j\in {\mathcal{A}}_i,\\ 0& \text{otherwise}.\end{array}$$

$$v^{{\pi }_1,{\pi }_2}\ge v^{{\pi }_1^{\prime },{\pi }_2}.$$

$$(x^{\pi }{)}_{\pi }=(I-\gamma P_{\pi }{)}^{-T}1,$$

$$(x^{\pi }{)}_a=0,\forall a\in \mathcal{A},a\notin \pi .$$

$$1^T{x}^{\pi }=1^T(I-\gamma P_{\pi }{)}^{-T}1=[(I-\gamma P_{\pi }{)}^{-1}1{]}^{T}1=\frac{1}{1-\gamma }{1}^{T}1=\frac{l}{1-\gamma }.$$

$$(x^{\pi }{)}_{\pi }-1=\gamma P_{\pi }(I-\gamma P_{\pi }{)}^{-T}1\ge 0.$$

$$1^T{v}^{\pi }=1^T(I-\gamma P_{\pi }{)}^{-1}{r}_{\pi }={\mathbf{r}}_{\pi }^T(I-\gamma P_{\pi }{)}^{-T}1={\mathbf{r}}_{\pi }^T(x^{\pi }{)}_{\pi }=(x^{\pi }{)}^{T}r,$$

$$v^{{\pi }^{\prime }}-v^{\pi }$$

$$=(I-\gamma P_{{\pi }^{\prime }}{)}^{-1}{\left[r-(J-\gamma P)v_{\pi }\right]}_{{\pi }^{\prime }}=(I-\gamma P_{{\pi }^{\prime }}{)}^{-1}(r^{\pi }{)}_{{\pi }^{\prime }};$$

$$1^T(v^{{\pi }^{\prime }}-v^{\pi })$$

$$(r^{{\pi }_1,{\pi }_2}{)}_{{\mathcal{A}}_1}\le 0,(r^{{\pi }_1,{\pi }_2}{)}_{{\mathcal{A}}_2}\ge 0.$$

$$v^{{\pi }_1^{\prime },{\pi }_2}-v^{{\pi }_1,{\pi }_2}=(I-\gamma P_{{\pi }_1^{\prime },{\pi }_2}{)}^{-1}(r^{{\pi }_1,{\pi }_2}{)}_{{\pi }_1^{\prime },{\pi }_2}=\sum _{n=0}^{\infty }{\gamma }^n{P}_{{\pi }_1^{\prime },{\pi }_2}^n(r^{{\pi }_1,{\pi }_2}{)}_{{\pi }_1^{\prime },{\pi }_2}\le 0,$$

$${\pi }_1^{\prime }(s_1)=\{\begin{array}{ll}{a}^{\prime }& \text{if }{s}_1=s;\\ {\pi }_1(s_1)& \text{else}.\end{array}$$

$$x_{a^{\prime }}^{{\pi }_1^{\prime },{\pi }_2}{r}_{a^{\prime }}^{\pi }=1^T(v^{{\pi }_1^{\prime },{\pi }_2}-v^{{\pi }_1,{\pi }_2})\le 0,$$

$$1^T\left(v^{{\pi }_1^{\ast },{\pi }_2^{\ast }}-v^{{\pi }_1^{n+M},{\pi }_2^{n+M}}\right)\le \frac{(1-\gamma {)}^2}{n^2}\cdot 1^T\left(v^{{\pi }_1^{\ast },{\pi }_2^{\ast }}-v^{{\pi }_1^n,{\pi }_2^n}\right).$$

$$1^T(v^{{\pi }^{\ast }}-v^{{\pi }^{n+1}})\le \left(1-\frac{1-\gamma }{l}\right)1^T(v^{{\pi }^{\ast }}-v^{{\pi }^n}).$$

$$1^T(v^{{\pi }^{n+1}}-v^{{\pi }^n})$$

$$\ge \frac{1-\gamma }{l}\sum _{a\in \mathcal{A}}{r}_a^{{\pi }^n}{x}_a^{{\pi }^{n+1}}=\frac{1-\gamma }{l}{1}^T(v^{{\pi }^{\ast }}-v^{{\pi }^n}),$$

$$1^T(v^{{\pi }^{\ast }}-v^{{\pi }^{n+1}})\le \gamma 1^T(v^{{\pi }^{\ast }}-v^{{\pi }^n}).$$

$$1^T(v^{{\pi }^{\ast }}-v^{{\pi }^{n+M}})$$

$$=\frac{(1-\gamma {)}^2}{l^2}{1}^T(v^{{\pi }^{\ast }}-v^{{\pi }^n}),$$

$$1^T(v^{{\pi }^{\ast }}-v^{{\pi }^{n+M}})\le \frac{(1-\gamma {)}^2}{l^2}{1}^T(v^{{\pi }^{\ast }}-v^{{\pi }^n}),$$

$${\pi }_1^{\prime }(s)=\{\begin{array}{ll}{a}_1,& \text{if }s=s_1,\\ {\pi }^n(s),& \text{others},\end{array}$$

$$1^T(v^{{\pi }^{\ast }}-v^{{\pi }^{\prime }})\le \left(1-\frac{1-\gamma }{n}\right)1^T(v^{\ast }-v^{{\pi }^n}).$$

$$1^T{v}^{{\pi }^{n+1}}\ge 1^T{v}^{{\pi }^{\prime }},$$

$$1^T(v^{{\pi }^{\ast }}-v^{{\pi }^{n+1}})\le \left(1-\frac{1-\gamma }{n}\right)1^T(v^{\ast }-v^{{\pi }^n}).$$

$$1^T(v^{{\pi }^{\ast }}-v^{{\pi }^{n+M}})\le \frac{(1-\gamma {)}^2}{l^2}{1}^T(v^{{\pi }^{\ast }}-v^{{\pi }^n}),$$

$$\mathcal{O}(Mm)$$

$$1^T(v^{{\pi }^{n+1}}-v^{{\pi }^n})=\sum _{a\in {\pi }_1^{n+1}}{x}_a^{{\pi }^{n+1}}{r}_a^{{\pi }^n}+\sum _{a\in {\pi }_2^{n+1}}{x}_a^{{\pi }^{n+1}}{r}_a^{{\pi }^n}\ge \sum _{a\in {\pi }_1^{n+1}}{x}_a^{{\pi }^{n+1}}{r}_a^{{\pi }^n}\ge 0,$$

$$1^T{v}^{{\pi }^n}\le 1^T{v}^{{\pi }^{n+1}}.$$

$$1^T(v^{{\pi }^{\ast }}-v^{{\eta }^n})=1^T(v^{{\pi }_1^{\ast },{\pi }_2^{\ast }}-v^{{\pi }_1^n,{\pi }_2^{\ast }})=-(x^{{\pi }_1^n,{\pi }_2^{\ast }}{)}^T{r}^{{\pi }_1^{\ast },{\pi }_2^{\ast }}=-(x^{{\pi }_1^n,{\pi }_2^{\ast }}{)}_{{\mathcal{A}}_1}^T{r}_{{\mathcal{A}}_1}^{{\pi }_1^{\ast },{\pi }_2^{\ast }}\ge 0,$$

Full text

1 Introduction

Markov decision process (MDP) is a widely used model in machine learning and operations research [1], which establishs basic rules of reinforcement learning. While solving an MDP focuses on maximizing (minimizing) the total reward (cost) for only one player, we consider a broader class of problems, the 2-player turn based stochastic games (2-TBSG) [15], which involves two players with opposite objectives. One player aims to maximize the total reward, and the other player aims to minimize the total reward. MDP and 2-TBSG have many useful applications, see [8, 3, 12, 2, 5, 10, 16].

Similar to MDP, every 2-TBSG has its state set and action set, both of which are divided into two subsets for each player, respectively. Moreover, its transition probability matrix describes the transition distribution over the state set conditioned on the current action, and its reward function describes the immediate reward when taking the action.

We use a strategy (policy) to denote a mapping from the state set into the action set. In our setting, we focus on the discounted 2-TBSG, where the reward in later steps is multiplied by a discounted factor. Given strategies (policies) for both players, the total reward is defined to be the sum of all discounted rewards. We solve a 2-TBSG by finding its Nash equilibrium strategy (equilibrium strategy for short), where the first player cannot change its own strategy to obtain a larger total reward, and the second player cannot change its own strategy to obtain a smaller total reward. MDP can be viewed as a special case of 2-TBSG, where all states belong to the first player. In such cases, the equilibrium strategy agrees with the optimal policy of MDP.

MDPs have their linear programming (LP) formulations [3]. Hence algorithms solving LP problems can be used to solve MDPs. One of the most commonly used algorithm in MDP is the policy iteration algorithm [8], which can be viewed as a parallel counterpart of the simplex method solving the corresponding LP. In paper [18], both the simplex method solving the corresponding LP and the policy iteration algorithms have been proved to find the optimal policy in $\mathcal{O}\left(\frac{ml}{1-\gamma}\log\frac{l}{1-\gamma}\right)$, where $m,l,\gamma$ are the number of actions, the number of states and the discounted factor, respectively. Later in [7], the bound for the policy iteration algorithm is improved by a factor $l$ to $\mathcal{O}\left(\frac{m}{1-\gamma}\log\frac{l}{1-\gamma}\right)$. In [14], this bound is improved to $\mathcal{O}\left(\frac{m}{1-\gamma}\log\frac{1}{1-\gamma}\right)$. When the MDP is deterministic (all transition probabilities are either [math] or $1$), a strongly polynomial bound independent on the discounted factor is proved in [11] for the simplex policy iteration method (each iteration changes only one action): $\mathcal{O}(m^{2}l^{3}\log^{2}l)$ for uniform discounted MDPs and $\mathcal{O}(m^{3}l^{5}\log^{2}l)$ for nonuniform discounted MDPs.

However, there is no simple LP formulation for 2-TBSGs. The strategy iteration algorithm [13], an analogue to the policy iteration, is a commonly used algorithm in finding the equilibrium strategy of 2-TBSGs. It is a strongly polynomial time algorithm first proved in [7] with a guarantee to find the equilibrium in $\mathcal{O}\left(\frac{m}{1-\gamma}\log\frac{l}{1-\gamma}\right)$ iterations if the discounted factor is fixed. When the discounted factor is not fixed, an exponential lower bound is given for the policy iteration in MDP [4] and for the strategy iteration in 2-TBSG [6]. It is an open problem whether there is a strongly polynomial algorithm whose complexity is independent of the discounted factor for 2-TBSG.

Motivated by the strongly polynomial simplex algorithm for solving MDPs, we present a simplex strategy iteration algorithm and a modified simplex strategy iteration algorithm for the 2-TBSG. In both algorithms each player updates in turn, where the second player always finds the best counterstrategy in its turn. In the simplex strategy iteration algorithm the first player updates its strategy using the simplex algorithm. In the modified simplex strategy iteration algorithm, the first player updates the action leading to the largest improvement after the second player finds the optimal counterstrategy. When the second player is trivial, the 2-TBSG becomes an MDP and the simplex strategy iteration algorithm can find its solution in strongly polynomial time independent of the discounted factor, which is a property not possessed by the strategy iteration algorithm in [7].

We also develop a proof technique to prove the strongly polynomial complexity for a class of geometrically converging algorithms. This class of algorithms includes the strategy iteration algorithm, the simplex strategy iteration algorithm, and the modified simplex strategy iteration algorithm. The complexity for the strategy iteration algorithm given in [7] can be recovered by our techniques. Our techniques use a combination of the current strategy and the equilibrium strategy. We establish a bound of ratio between the difference of value from the current strategy to the equilibrium strategy, and the difference of value from the combined strategy to the equilibrium strategy. Using this bound and the geometrically converging property, we can prove that after a certain number of iterations, one action will disappear forever, which leads to strongly polynomial convergence when the discount factor is fixed. Although we have not fully answered the open progblem, our algorithms and analysis point out a possible way for conquering the difficulities.

Furthermore, 2-TBSG where each state has exactly two actions can be transformed into a linear complementary problem [9]. An MDP where each state has exactly two actions can be solved by a combinatorial interior point method [17]. In this paper we present a way to transform a general 2-TBSG into a 2-TBSG where each state has exactly two actions. The number of states in this constructed 2-TBSG is $\tilde{\mathcal{O}}(m+l)$ (we use $\tilde{\mathcal{O}}$ to hide log factors of $l,m$). This result enables the application of both results in [9, 17] to general cases.

The rest of this paper is organized as follows. In Section 2 we present some basic concepts and lemmas of the 2-TBSG. In Section 3 we describe the simplex strategy iteration algorithm and the modified simplex strategy iteration algorithm. The proof of complexity of the class of geometrically converging algorithm is given in Section 4. The transformation from general 2-TBSGs into special 2-TBSGs is introduced in Section 5.

2 Preliminaries

In this section, we present some basic concepts of 2-TBSG. Our focus here is on the discounted 2-TBSG, defined as follows.

Definition 2.1**.**

A discounted 2-TBSG (2-TBSG for short) consists of a tuple $(\mathcal{S},\mathcal{A},P,r,\gamma)$, where $\mathcal{S}=\mathcal{S}_{1}\cup\mathcal{S}_{2},\mathcal{A}=\mathcal{A}_{1}\cup\mathcal{A}_{2}$. $\mathcal{S}_{1},\mathcal{S}_{2},\mathcal{A}_{1},\mathcal{A}_{2}$ are the state set and the action set of each player, respectively. $P\in\mathbb{R}^{|\mathcal{A}|\times|\mathcal{S}|}$ is the transition probability matrix, where $P(a,s)$ denotes the probability of the event that the next state is $s$ conditioned on the current action $a$. $r\in\mathbb{R}^{|\mathcal{A}|}$ is the reward vector, where $r_{a}$ denotes the immediate reward function received using action $a$. To be convenient, we use $m=|\mathcal{A}|$ to denote the number of actions, and $l=|\mathcal{S}|$ to denote the number of states.

Given a state $s\in\mathcal{S}$ in 2-TBSG setting, we use $\mathcal{A}_{s}$ to denote the set of available actions corresponding to state $s$. A deterministic strategy (strategy for short) $\pi=(\pi_{1},\pi_{2})$ is defined such that $\pi_{1},\pi_{2}$ are mappings from $\mathcal{S}_{1}$ to $\mathcal{A}_{1}$ and from $\mathcal{S}_{2}$ to $\mathcal{A}_{2}$, respectively. Moreover, each state $s\in\mathcal{S}$ matches to an action in $\mathcal{A}_{s}$.

For a given strategy $\pi=(\pi_{1},\pi_{2})$, we define the transition probability matrix $P_{\pi}\in\mathbb{R}^{l\times l}$ and reward function $r_{\pi}\in\mathbb{R}^{l}$ with respect to $\pi$. The $i$-th row of $P_{\pi}$ is chosen to be the row of action $\pi(i)$ in $P$, and the $i$-th element of $r_{\pi}$ is chosen to be the reward of action $\pi(i)$. It is easy to observe that the matrix $P_{\pi}$ is a stochastic matrix. We next define the value vector and the modified reward function.

Definition 2.2**.**

The value vector $v^{\pi}\in\mathbb{R}^{l}$ of a given strategy $\pi=(\pi_{1},\pi_{2})$ is

[TABLE]

Definition 2.3**.**

The modified reward function $r^{\pi}\in\mathbb{R}^{m}$ of a given strategy $\pi$ is defined as

[TABLE]

where $J\in\mathbb{R}^{m\times l}$ is defined as

[TABLE]

Furthermore, for a given 2-TBSG, the optimal counterstrategy against another player’s given strategy is defined in Definition 2.4. The equilibrium strategy is given in Definition 2.5.

Definition 2.4**.**

For player 2’s strategy $\pi_{2}$, player 1’s strategy $\pi_{1}$ is the optimal counterstrategy against $\pi_{2}$ if and only if for any strategy $\pi_{1}^{\prime}$ of player 1, we have

[TABLE]

Player 2’s optimal counterstrategy can be defined similarly: $\pi_{2}$ is the optimal counterstrategy against $\pi_{1}$ if and only if for any strategy $\pi_{2}^{\prime}$, $v^{\pi_{1},\pi_{2}}\leq v^{\pi_{1},\pi_{2}^{\prime}}$. Here for two value vector $v,v^{\prime}$, we say $v\geq v^{\prime}$ ($v\leq v^{\prime}$) if and only if $v(s)\geq v^{\prime}(s)$ ($v(s)\leq v^{\prime}(s)$) for $\forall s\in\mathcal{S}$.

Definition 2.5**.**

A strategy $\pi=(\pi_{1},\pi_{2})$ is called an equilibrium strategy, if and only if $\pi_{1}$ is the optimal counterstrategy against $\pi_{2}$, and $\pi_{2}$ is the optimal counterstrategy against $\pi_{1}$.

To describe the property of equilibrium strategies, we present Theorems 2.6 and 2.7 given in [7, 15]. Theorem 2.6 indicates the existence of an equilibrium strategy.

Theorem 2.6**.**

Every 2-TBSG has at least an equilibrium strategy. If $\pi$ and $\pi^{\prime}$ are two equilibrium strategies, then $v^{\pi}=v^{\pi^{\prime}}$. Furthermore, for any player 1’s strategy $\pi_{1}$ (or player 2’s strategy $\pi_{2}$), there always exists a player 2’s optimal counterstrategy $\pi_{2}$ against $\pi_{1}$ (player 1’s optimal counterstrategy $\pi_{1}$ against $\pi_{2}$), and for any two optimal counterstrategy $\pi_{2},\pi_{2}^{\prime}$ ($\pi_{1},\pi_{1}^{\prime}$), we have $v^{\pi_{1},\pi_{2}}=v^{\pi_{1}^{\prime},\pi_{2}}$ ($v^{\pi_{1},\pi_{2}}=v^{\pi_{1},\pi_{2}^{\prime}}$).

The next theorem points out a useful depiction of the value function at the equilibrium.

Theorem 2.7**.**

Let $\pi^{*}$ be an equilibrium strategy for 2-TBSG. If $\pi_{1}$ is a strategy of player 1, and $\pi_{2}$ is player 2’s optimal counterstrategy against $\pi_{1}$, then we have $v^{\pi^{*}}\geq v^{\pi_{1},\pi_{2}}$. The equality holds if and only if $(\pi_{1},\pi_{2})$ is an equilibrium strategy.

We now define the flux vector of a given strategy $\pi$.

Definition 2.8**.**

The flux $x^{\pi}\in\mathbb{R}^{m}$ of a given strategy $\pi$ is defined as

[TABLE]

Our next lemma presents bounds and conditions of the flux vector, and the relationship among the value function, the flux vector and reduced costs. This lemma and the following several lemmas can be found in [7]. To make the paper self-contained, we briefly give their proofs.

Lemma 2.9**.**

For any strategy $\pi$, we have

$\mathbf{1}^{T}x^{\pi}=\frac{l}{1-\gamma}$; 2. 2.

for any $a\in\pi$, $1\leq x_{a}\leq\frac{l}{1-\gamma}$; 3. 3.

$\mathbf{1}^{T}v^{\pi}=(x^{\pi})^{T}r$; 4. 4.

$v^{\pi^{\prime}}-v^{\pi}=(I-\gamma P_{\pi^{\prime}})^{-1}(r^{\pi})_{\pi^{\prime}}$, and moreover, $\mathbf{1}^{T}(v^{\pi^{\prime}}-v^{\pi})=(x^{\pi^{\prime}})^{T}r^{\pi}$.

Proof.

Item (1) is proved by

[TABLE]

Item (2) is due to

[TABLE]

This indicates that $(x^{\pi})_{a}\geq 1$, $\forall a\in\pi$. Hence we have $x^{\pi}\geq 0$ and $(x^{\pi})_{a}\leq\frac{l}{1-\gamma}$ from item (1). Finally the last two items are obtained from

[TABLE]

and

[TABLE]

∎

In the following, we present a lemma indicating the positiveness or negativeness of the reduced costs of optimal counterstrategies and equilibrium strategies.

Lemma 2.10**.**

A strategy $\pi_{1}$ for player 1 is an optimal counterstrategy against player 2’s strategy $\pi_{2}$ if only if $(r^{\pi_{1},\pi_{2}})_{\mathcal{A}_{1}}\leq 0$. 2. 2.

A strategy $\pi_{2}$ for player 2 is an optimal counterstrategy against player 1’s strategy $\pi_{1}$ if only if $(r^{\pi_{1},\pi_{2}})_{\mathcal{A}_{2}}\geq 0$. 3. 3.

A strategy $\pi=(\pi_{1},\pi_{2})$ is an equilibrium strategy if and only if it satisfies:

[TABLE]

Proof.

If $\pi_{1},\pi_{2}$ satisfies $(r^{\pi_{1},\pi_{2}})_{\mathcal{A}_{1}}\leq 0$, then for any player 1’s strategy $\pi_{1}^{\prime}$, we have

[TABLE]

where the last inequality follows from $(r^{\pi_{1},\pi_{2}})_{\pi_{1}^{\prime}}\leq 0$ for $\pi_{1}^{\prime}\in\mathcal{A}_{1}$ and $(r^{\pi_{1},\pi_{2}})_{\pi_{2}}=0$.

Suppose that player 1’s strategy $\pi_{1}$ is the optimal counterstrategy against player 2’s strategy $\pi_{2}$. For any $a^{\prime}\in\mathcal{A}_{s}$, $s\in\mathcal{S}_{1}$ and $a^{\prime}\not\in\pi_{1}$, we let

[TABLE]

Then again from Lemma 2.9 (4), we have

[TABLE]

where the inequality comes from the definition of equilibrium strategies. Since $a^{\prime}\in\pi_{1}^{\prime}$, we have $x^{\pi^{\prime}_{1},\pi_{2}}_{a^{\prime}}\geq 1$, which indicates that $r^{\pi}_{a^{\prime}}\leq 0$. With this estimation and $r^{\pi}_{a}=0$ for $\forall a\in\pi$, we have proved that $(r^{\pi})_{\mathcal{A}_{1}}\leq 0$. Hence, item (1) is established, and the proof of item (2) is similar. Finally item (3) follows from items (1) and (2) directly. ∎

3 Geometrically Converging Algorithms

Inspired by the simplex method solving the LP corresponding to the MDP and the strategy iteration algorithm given in [7], we propose a simplex strategy iteration (Algorithm 1) and a modified simplex strategy iteration algorithm (Algorithm 2) for 2-TBSG.

The simplex strategy iteration algorithm can be viewed as a generalization of the strongly polynomial simplex algorithm in solving MDPs [11]. In our algorithm, both players update their strategies in turn. In each iteration, while the first player updates its strategy using the simplex method, which means only updating the action with the largest reduced cost, the second player updates its strategy according to the optimal counterstrategy. When the second player has only one possible action and the transition matrix is deterministic, the 2-TBSG reduces to a deterministic MDP. Then the simplex strategy iteration algorithm can find an equilibrium (optimal) strategy in strongly polynomial time independent of $\gamma$, which is a property has not been proven for the strategy iteration [7].

As for the modified simplex strategy iteration algorithm, it can be viewed as a modification of the simplex strategy iteration algorithm. In this algorithm, both players also update their strategies in turn, and the second player always finds the optimal counterstrategies in its moves. However, in each of the first player’s move, only the action is updated which leads to the biggest improvement on the value function when the second player uses the optimal counterstrategy.

It is easy to know that every iteration of the simplex strategy iteration algorithm involves a step of a simplex update and a solution to an MDP. And every iteration of the modified simplex strategy iteration algorithm involves solutions to multiple MDPs. Hence every iteration in both of these two algorithms can be solved in strongly polynomial time when the discounted factor is fixed.

Next we present a class of geometrically converging algorithms used for proving the strongly polynomial complexity for several algorithms in the next section.

Definition 3.1**.**

We say a strategy-update algorithm (algorithms which update strategies for both players in each iteration) is a geometrically converging algorithm with parameter a $M$, if it updates a strategy $\pi^{n}=(\pi_{1}^{n},\pi_{2}^{n})$ to $\pi^{n+1}=(\pi_{1}^{n+1},\pi_{2}^{n+1})$ such that the following properties holds.

• •

$\pi_{2}^{n+1}$ is the optimal counterstrategy against $\pi^{n+1}_{1}$;

• •

$(r^{\pi^{n}})_{\pi_{1}^{n+1}}\geq 0$;

• •

If $\mathbf{1}^{T}(v^{\pi^{n+1}}-v^{\pi^{n}})=0$, then $\pi^{n}$ is an equilibrium strategy;

• •

The updates of this algorithm satisfies

[TABLE]

To begin with, we exhibit a lemma indicating the geometrically converging property of the value function in the simplex strategy iteration algorithm.

Lemma 3.2**.**

Suppose the sequence of strategy generated by the simplex strategy iteration algorithm is $\pi^{1}=(\pi^{1}_{1},\pi^{1}_{2}),\pi^{2}=(\pi^{2}_{1},\pi^{2}_{2})\cdots,\pi^{n}=(\pi^{n}_{1},\pi^{n}_{2}),\cdots$. Then the following inequality holds

[TABLE]

Proof.

According to Algorithm 1, we have

[TABLE]

where the second and third inequalities follow from Lemma 2.9 (2) and the choice of $a_{1}=\arg\max_{a\in\mathcal{A}_{1}}r^{\pi^{n}}_{a}$, the fourth inequality follows from Lemma 2.10, and the first inequality and last equation are due to Lemma 2.9 (4) and Lemma 2.10. ∎

Using this lemma, we show in the next proposition that the strategy iteration algorithm, Algorithm 1 and Algorithm 2 all belong to the class of geometrically converging algorithms.

Proposition 3.3**.**

The strategy iteration algorithm given in **[7]** is a geometrically converging algorithm with parameter $M=\mathcal{O}\left(\frac{1}{1-\gamma}\log\frac{l}{1-\gamma}\right)$; 2. 2.

The simplex strategy iteration algorithm (Algorithm 1) is a geometrically converging algorithm with parameter $M=\mathcal{O}\left(\frac{l}{1-\gamma}\log\frac{l}{1-\gamma}\right)$; 3. 3.

The modified simplex strategy iteration algorithm (Algorithm 2) is a geometrically converging algorithm with parameter $M=\mathcal{O}\left(\frac{l}{1-\gamma}\log\frac{l}{1-\gamma}\right)$;

Proof.

It is easy to verify that the previous described three algorithms satisfy the first three conditions in the definition of geometrically converging algorithms. Next, we prove that all of these algorithms satisfy the last condition. For the strategy iteration algorithm, according to Lemma 4.8 and Lemma 5.4 given in [7], we have

[TABLE]

Hence if $M=\frac{2c_{1}}{1-\gamma}\log\frac{l}{1-\gamma}=\mathcal{O}\left(\frac{1}{1-\gamma}\log\frac{l}{1-\gamma}\right)$ ($c_{1}\geq 1$ is a constant), then we obtain

[TABLE]

and the last condition of geometrically converging algorithms is verified.

For the simplex strategy iteration algorithm, if we choose $M=\mathcal{O}\left(\frac{l}{1-\gamma}\log\frac{l}{1-\gamma}\right)$ ($c_{2}\geq 1$ is a constant), then according to inequality (1) we have

[TABLE]

and the last condition of geometrically converging algorithms is verified.

Finally we consider the modified simplex strategy iteration algorithm. For $n\geq 2$, let $a_{1}=\arg\max_{a\in\mathcal{A}_{1}}r^{\pi^{n}}_{a}$, where $a_{1}$ is an action of state $s_{1}$. Let

[TABLE]

$\pi_{2}^{\prime}$ be player 2’s optimal counterstrategy against $\pi_{1}^{\prime}$, and $\pi^{\prime}=(\pi_{1}^{\prime},\pi_{2}^{\prime})$. Then from inequality (1), we have

[TABLE]

According to Algorithm 2, we have

[TABLE]

which leads to the following estimation:

[TABLE]

Therefore, similar to the previous case we can choose $M=\mathcal{O}\left(\frac{l}{1-\gamma}\log\frac{l}{1-\gamma}\right)$ such that

[TABLE]

and the last condition of geometrically converging algorithms is verified. ∎

4 Strongly Polynomial Complexity of Geometrically Converging Algorithms

In this section, we develop the strongly polynomial property of geometric converging algorithms if the parameter $M$ is viewed as a constant. Slightly different from the proof in [7] for the strategy $(\pi_{1}^{n},\pi_{2}^{n})$ at the $n$-th iteration, we present a proof by considering the strategy $(\pi_{1}^{n},\pi_{2}^{*})$, where $(\pi_{1}^{*},\pi_{2}^{*})$ is an equilibrium strategy. We show that $\mathbf{1}^{T}(v^{\pi_{1}^{*},\pi_{2}^{*}}-v^{\pi_{1}^{n},\pi_{2}^{*}})$ can be both upper and lower bounded by some proportion of $\mathbf{1}^{T}(v^{\pi_{1}^{*},\pi_{2}^{*}}-v^{\pi_{1}^{n},\pi_{2}^{n}})$. By applying the property of geometrically converging algorithms, we obtain that after a certain number of iterations, a player 1’s action will disappear in $\pi_{1}^{n}$ forever.

Theorem 4.1**.**

Any geometrically converging algorithm with a parameter $M$ finds the equilibrium strategy in

[TABLE]

number of iterations.

Proof.

Suppose $\pi^{1}=(\pi_{1}^{1},\pi_{2}^{1}),\pi^{2}=(\pi_{1}^{2},\pi_{2}^{2}),\cdots,\pi^{n}=(\pi_{1}^{n},\pi_{2}^{n})$ is the sequence generated by a geometrically converging algorithm. We define $\eta^{n}=(\pi_{1}^{n},\pi_{2}^{*})$, where $\pi^{*}=(\pi_{1}^{*},\pi_{2}^{*})$ is one of the equilibrium strategy.

According to Lemma 2.10 and the fact that $\pi_{2}^{n+1}$ is the optimal counterstrategy against $\pi_{1}^{n+1}$, and the definition of geometrically converging algorithm, we have

[TABLE]

which directly leads to

[TABLE]

According to Lemma 2.10, we have

[TABLE]

which implies

[TABLE]

We next prove the following inequality:

[TABLE]

A direct calculation gives

[TABLE]

where the last inequality is obtained from Lemma 2.10. Then noticing that

[TABLE]

we have

[TABLE]

Then the inequality (4) is proved.

Finally, we prove that for any $n$, either there exists an action $a_{1}$ in $\pi_{1}^{n}$ will never belong to $\pi_{1}^{n+m}$ when $m>M$, or we have

[TABLE]

Actually for any $p>M$, suppose $\mathbf{1}^{T}(v^{\pi^{n+M+1}}-v^{\pi^{n+M}})\neq 0$, we obtain

[TABLE]

from (2) and the definition of geometrically converging algorithm. Hence according to (3) and (4), we get

[TABLE]

Therefore, choosing $a_{1}=\arg\min_{a\in\pi_{1}^{n}}r_{a}^{\pi^{*}}\leq 0$, and because for any $a\in\pi_{1}^{n}$, $r_{a}^{\pi^{*}}\leq 0$ according to Lemma 2.10, we obtain

[TABLE]

from Lemma 2.9. If $a\in\pi_{1}^{n+p}$, we have

[TABLE]

where the first inequality is due to Lemma 2.10 and the second inequality is due to Lemma 2.9. Therefore, combining these two inequalities and the inequality (5) and noticing that $r_{a_{1}}^{\pi^{*}}\leq 0$, we get

[TABLE]

This leads to contradiction.

The previous derivation means that if $\mathbf{1}^{T}(v^{\pi^{n+M+1}}-v^{\pi^{n+M}})=0$ does not hold for $n$, then an action of $\pi^{n}$ must disappear after $\pi^{n+M}$ forever. Hence every after $M$ iterations an action will disappear forever. This process cannot happen for more than $m-l$ times (since there are $m$ actions and every strategy has $n$ actions), which indicates that for some $n>M(m-l)$,

[TABLE]

It follows from the definition of geometrically converging algorithm that $\pi^{n+M}$ is the equilibrium strategy. This indicates that within

[TABLE]

number of iterations, we can find one of the equilibrium strategies. ∎

Our next theorem presents the complexity of the strategy iteration algorithm, the simplex strategy iteration algorithm and the modified simplex strategy iteration algorithm.

Theorem 4.2**.**

The following algorithms has strongly polynomial convergence when the discounted factor is fixed.

• •

The strategy iteration algorithm given in **[7]** can find the equilibrium strategy within $\mathcal{O}\left(\frac{m}{1-\gamma}\log\frac{l}{1-\gamma}\right)$ iterations;

• •

The simplex strategy iteration algorithm (Algorithm 1) can find the equilibrium strategy within $\mathcal{O}\left(\frac{ml}{1-\gamma}\log\frac{l}{1-\gamma}\right)$ iterations;

• •

The modified simplex strategy iteration algorithm (Algorithm 2) can find the equilibrium strategy within $\mathcal{O}\left(\frac{ml}{1-\gamma}\log\frac{l}{1-\gamma}\right)$ iterations;

Proof.

The proof of this theorem directly follows from Theorem 4.1 and Proposition 3.3. ∎

Remark 1*.*

It is easy to note that the terminated condition of the simplex strategy iteration algorithm and the modified simplex strategy iteration algorithm is equivalent to the condition of meeting an equilibrium strategy. Hence the above theorem also indicates that these two algorithms terminate within $\mathcal{O}\left(\frac{ml}{1-\gamma}\log\frac{l}{1-\gamma}\right)$ iterations.

5 Transform General 2-TBSGs into Special 2-TBSGs

We prove in this section that every 2-TBSG can be transformed into a new 2-TBSG where each state has exactly two actions. A formal description is given in the next theorem.

Theorem 5.1**.**

Given a 2-TBSG with $l$ states and $m$ actions whose state set is $\mathcal{S}$, we can construct a new 2-TBSG with state set $\mathcal{S}^{\prime}$ satisfying the following properties.

• •

The number of states in the constructed 2-TBSG is bounded by a polynomial of $m$ and $l$:

[TABLE]

• •

$\mathcal{S}\subset\mathcal{S}^{\prime}$* and the value function $V$ at the equilibrium of the constructed 2-TBSG satisfies:*

[TABLE]

where $v$ is the equilibrium value function of the original 2-TBSG, and

[TABLE]

Proof.

Our proof consists of two parts. In the first part, we construct a new 2-TBSG where each state has no more than two actions, and the value function at equilibrium of original 2-TBSG can be easily obtained given the equilibrium value of the constructed 2-TBSG (proportional to the value at some states in the constructed 2-TBSG). In the second part, we modify the constructed 2-TBSG so that each state has exactly two actions, while keeping the equilibrium value unchanged by constructing an obvious undesirable action for those states with only one action.

We first construct a binary tree rooted at $s$ with exactly $|A_{s}|$ leaves, and the depth of the tree is exactly $p=\lceil\log m\rceil$. This tree is called the depth-$p$ binary tree of state $s$:

• •

In the first $p-\log\lceil|A_{s}|\rceil$ layers, each node has only one child.

• •

In the last $\log\lceil|A_{s}|\rceil$ layers, it is a binary tree with exactly $|A_{s}|$ leaves.

• •

Every leaves has depth $p$.

Each node except the root $s$ and all leaves in the depth-$p$ binary tree of $s$ are assigned with a new state whose owner is same as state $s$ (player 1 or player 2). We use $S_{1}$ to denote the set of states in the first $p-2$ layers, and $S_{2}$ to denote the set of states in the $(p-1)$-th layer. The parameters (transition probabilities, rewards, discounted factor) are given as follows:

• •

For each state in $S_{1}$, one or two actions are assigned to it depending on how many children states (its children in the binary tree) it has, with probability 1 leading to a child state and reward 0.

• •

For set $S_{2}$, each of their children nodes is assigned with an action of $s$ in the original 2-TBSG. This can be done since the total number of children nodes of $S_{2}$ is exactly $|\mathcal{A}_{s}|$. For each state in $S_{2}$, its actions are given by its children nodes. The transition probability and reward of taking that action is assigned to be the same as in the original 2-TBSG.

• •

The discounted factor in the constructed 2-TBSG is given by $\delta=\gamma^{1/p}$.

A special case of can be viewed in Figure 1 when $p=4,|\mathcal{A}_{s}|=7$.

It is easy to obtain that the number of states in the constructed 2-TBSG is no more than

[TABLE]

We next present a definition of final actions and the executing path of a state.

Definition 5.2**.**

For a given strategy $\pi^{\prime}$ in the constructed 2-TBSG cases and $s\in\mathcal{S}$, we continue the following process:

• •

$s_{0}\leftarrow s,\quad i\leftarrow 0$;

• •

If $\pi^{\prime}(s_{n})$ is a constructed action (not an action in the original 2-TBSG), then we let $s_{n+1}$ to be the state obtained by executing action $\pi^{\prime}(s_{n})$. Since all constructed actions are deterministic, there is only one choice of $s_{n+1}$. Then let $n\leftarrow n+1$.

• •

If $\pi^{\prime}(s_{n})$ is an action in the original 2-TBSG, then we stop this process, and call $\pi^{\prime}(s_{n})\in\mathcal{A}$ to be the final action of $s$, and path $s_{0}\to s_{1}\to\cdots\to s_{n}\to\pi^{\prime}(s_{n})$ to be the executed path from $s$ to action $\pi^{\prime}(s_{n})$.

Notice that the previous described process must be ended in $p-1$ steps, and all states in the executed path of $s$ must lie in the depth-$p$ binary tree of $s$. For any state $s\in\mathcal{S}$ and $a\in\mathcal{A}_{s}$, there exists a unique executed path from $s$ to $a$. In Figure 1 we present an example of final actions and executed paths. When the strategy $\pi^{\prime}$ follows bold arrows, the final action of $s$ will be $a_{3}$, and the executed path from $s$ to $a_{3}$ is $s\to s_{1}\to s_{2}\to s_{5}\to a_{3}$.

Based on the final actions, we define the corresponding strategy $\pi$ with respect to $\pi^{\prime}$ in the original 2-TBSG: for each state $s\in\mathcal{S}$, $\pi(s)$ is defined to be the final action of $s$ in $\pi^{\prime}$. Next, we prove that for any state $s\in\mathcal{S}$, the value of $s$ in strategy $\pi^{\prime}$ agrees with $\delta^{p-1}$ times the value of $s$ in strategy $\pi$. Actually, along the trajectory of $\pi^{\prime}$, we meet a final action every $p$ steps, and only final actions have nonzero rewards. Hence values of $s$ in $\pi$ and $\pi^{\prime}$ satisfy

[TABLE]

where $a^{\prime}$ denotes actions along strategy $\pi^{\prime}$, and $a$ denotes actions along strategy $\pi$.

What is left in the proof is to show that if $\pi^{\prime}=(\pi_{1}^{\prime},\pi_{2}^{\prime})$ is an equilibrium strategy in the constructed 2-TBSG, then $\pi=(\pi_{1},\pi_{2})$ is an equilibrium strategy in the original 2-TBSG. For any player 1’s state $s$ and action $a\in\mathcal{A}_{s}$, we use $\eta=(\eta_{1},\pi_{2})$ to denote the strategy in the original 2-TBSG:

[TABLE]

In the constructed 2-TBSG, there exists a unique executed path $T$ from $s$ to action $a$, and for any state $s_{1}$ on this path $T$, there is only one action $\tau(s_{1})$ in $\mathcal{A}^{\prime}_{s_{1}}$ such that the next state when using $\tau(s_{1})$ also lies on $T$. We define player 1’s strategy $\eta^{\prime}=(\eta^{\prime}_{1},\pi_{2})$ as follows:

[TABLE]

where $\lambda(s_{1})$ can be chosen $\mathcal{A}^{\prime}_{s_{1}}$ arbitrarily. Then it is easy to examine that $\eta$ is the corresponding strategy of $\eta^{\prime}$. Since $\eta$ is an equilibrium strategy of the constructed 2-TBSG, we have

[TABLE]

where the inequality is due to the property of equilibrium strategy. Furthermore, according to Lemma 2.9 and Lemma 2.10, we have $x_{a}^{\eta}r^{\pi}_{a}=\mathbf{1}^{T}(v^{\eta}-v^{\pi})\geq 0$, where $x_{a}^{\eta}\geq 1$. This indicates that $r^{\pi}_{a}\leq 0$. Since $a$ can be chosen arbitrarily, we have $(r^{\pi})_{\mathcal{A}_{1}}\leq 0$, and similarly $(r^{\pi})_{\mathcal{A}_{2}}\geq 0$. Again according to Lemma 2.10, we obtain that $\pi$ is an equilibrium strategy of the original 2-TBSG.

Next, we handle states with only one actions. For each of such state, our technique is to construct another action which is obviously unacceptable to appear in equilibrium strategies. For state $s$ with only one action $a_{1}$, we assign it with another action $a_{2}$:

• •

The transition probability using action $a_{2}$ is identical to $a_{1}$.

• •

If state $s$ belongs to player 1, then the reward of $a_{2}$ is assigned to be smaller than the reward of $a_{1}$.

• •

If state $s$ belongs to player 2, then the reward of $a_{2}$ is assigned to be larger than the reward of $a_{1}$.

If we construct actions in such ways, it is obvious that action $a_{2}$ is inferior to $a_{1}$ according to its owner (player 1 or player 2). Hence any strategy which possesses $a_{2}$ is not an equilibrium strategy, since switching action $a_{2}$ into $a_{1}$ leads to a better strategy for its owner. Combining these two parts together proves Theorem 5.1. ∎

Remark 2*.*

Since it is easy to obtain the equilibrium strategy from the equilibrium value and vice versa, we can solve the original 2-TBSG by solving the constructed 2-TBSG.

6 Conclusion

In this paper, we propose two different algorithms for 2-TBSG with strongly polynomial complexity: the simplex strategy iteration algorithm and the modified simplex strategy iteration algorithm. We propose a class of geometrically converging algorithms and develop a proof technique to prove the strongly polynomial complexity when the discounted factor is fixed. Furthermore, we present how to transform a general 2-TBSG into a special 2-TBSG where each state has exactly two actions. Specifically, our simplex strategy iteration algorithm is coincident with the simplex method in the MDP cases.These analysis and properties shed some light on the open problem of solving the deterministic 2-TBSG in strongly polynomial time independent of the discount factor.