Entropy-Regularized Stochastic Games

TL;DR

This paper introduces entropy-regularized stochastic games, incorporating a regularization term to balance rationality and environmental uncertainty, and provides algorithms for computing optimal strategies with practical applications.

Contribution

It extends stochastic game theory by adding entropy regularization, proving existence of game values, and developing convex optimization algorithms for strategy computation.

Findings

Existence of value in entropy-regularized N-stage and discounted stochastic games.

Markovian and stationary strategies suffice to attain the value.

Numerical example demonstrates the impact of regularization on payoff.

Abstract

In two-player zero-sum stochastic games, where two competing players make decisions under uncertainty, a pair of optimal strategies is traditionally described by Nash equilibrium and computed under the assumption that the players have perfect information about the stochastic transition model of the environment. However, implementing such strategies may make the players vulnerable to unforeseen changes in the environment. In this paper, we introduce entropy-regularized stochastic games where each player aims to maximize the causal entropy of its strategy in addition to its expected payoff. The regularization term balances each player's rationality with its belief about the level of misinformation about the transition model. We consider both entropy-regularized $N$-stage and entropy-regularized discounted stochastic games, and establish the existence of a value in both games. Moreover, we prove the sufficiency of Markovian and stationary mixed strategies to attain the value, respectively, in $N$-stage and discounted games. Finally, we present algorithms, which are based on convex optimization problems, to compute the optimal strategies. In a numerical example, we demonstrate the proposed method on a motion planning scenario and illustrate the effect of the regularization term on the expected payoff.

Equations

$${\mu }_{t+1}(x^{t+1},u^t,w^t)=\mathcal{P}(x_{t+1}|x_t,u_t,w_t){\sigma }_t(u_t|h_t)$$

$$\times {\tau }_t(w_t|h_t){\mu }_t(h_t)$$

$$J(X^N,U^N,W^N):=$$

$$\sum _{t=1}^N{\mathbb{E}}^{{\overline{\mu }}_t}\mathcal{R}(X_t,U_t,W_t)+{\mathbb{E}}^{{\mu }_{T+1}}\mathcal{R}(X_{N+1})$$

$$\sum _{t=1}^{\infty }{\gamma }^{t-1}{\mathbb{E}}^{{\overline{\mu }}_t}\mathcal{R}(X_t,U_t,W_t).$$

$${\mathcal{V}}^{\star }=\underset{\mathbf{\sigma }\in {\Gamma }_u}{\max }\underset{\mathbf{\tau }\in {\Gamma }_w}{\min }f(\mathbf{\sigma },\mathbf{\tau })=\underset{\mathbf{\tau }\in {\Gamma }_w}{\min }\underset{\mathbf{\sigma }\in {\Gamma }_u}{\max }f(\mathbf{\sigma },\mathbf{\tau }).$$

$$H(X^N||Y^N,Z^N):=\sum _{t=1}^{N}H(X_t|X^{t-1},Y^t,Z^t),$$

$$H(X_t|X^{t-1},Y^t,Z^t):=$$

$$-\sum _{{\mathcal{X}}^N,{\mathcal{Y}}^N,{\mathcal{Z}}^N}\text{Pr}(x^t,y^t,z^t)\log \text{Pr}(x_t|x^{t-1},y^t,z^t).$$

$${\Phi }_N(\mathbf{\sigma },\mathbf{\tau }):=$$

$$-\frac{1}{{\beta }_2}H(W^N||X^N,U^{N-1}),$$

$${\Phi }_{\infty }(\mathbf{\sigma },\mathbf{\tau }):=$$

$$\displaystyle+\frac{1}{\beta_{1}}H(U_{t}|H_{t})-\frac{1}{\beta_{2}}H(W_{t}|H_{t})\Big{]},$$

$$\underset{\mathbf{\sigma }\in {\Gamma }_u^M}{\max }\underset{\mathbf{\tau }\in {\Gamma }_w^M}{\min }{\Phi }_N(\mathbf{\sigma },\mathbf{\tau })=\underset{\mathbf{\sigma }\in {\Gamma }_u}{\max }\underset{\mathbf{\tau }\in {\Gamma }_w}{\min }{\Phi }_N(\mathbf{\sigma },\mathbf{\tau }),$$

$$\underset{\mathbf{\tau }\in {\Gamma }_w^M}{\min }\underset{\mathbf{\sigma }\in {\Gamma }_u^M}{\max }{\Phi }_N(\mathbf{\sigma },\mathbf{\tau })=\underset{\mathbf{\tau }\in {\Gamma }_w}{\min }\underset{\mathbf{\sigma }\in {\Gamma }_u}{\max }{\Phi }_N(\mathbf{\sigma },\mathbf{\tau }).$$

$$\displaystyle\mathcal{V}_{t}^{\sigma_{t},\tau_{t}}(x_{t}):=\mathbb{E}^{\sigma_{t},\tau_{t}}\Big{[}$$

$$\displaystyle+\frac{1}{\beta_{2}}\log\tau_{t}(w_{t}|x_{t})\Big{]}$$

$$\underset{{\sigma }_t\in \Delta (\mathcal{U})}{\max }\underset{{\tau }_t\in \Delta (\mathcal{W})}{\min }{\mathcal{V}}_t^{{\sigma }_t,{\tau }_t}(x_t)=\underset{{\tau }_t\in \Delta (\mathcal{W})}{\min }\underset{{\sigma }_t\in \Delta (\mathcal{U})}{\max }{\mathcal{V}}_t^{{\sigma }_t,{\tau }_t}(x_t).$$

$${\sigma }_t^{\star }(u_t|x_t)\in$$

$$\displaystyle\arg\max_{\sigma_{t}\in\Delta(U)}\Big{[}-\frac{1}{\beta_{1}}\sum_{u_{t}\in\mathcal{U}}\sigma_{t}(u_{t}|x_{t})\log\sigma_{t}(u_{t}|x_{t})$$

$$\displaystyle-\frac{1}{\beta_{2}}\log\sum_{w_{t}\in\mathcal{W}}\exp\big{(}-\beta_{2}\sum_{u_{t}\in\mathcal{U}}\sigma_{t}(u_{t}|x_{t})\rho_{t}(x_{t},u_{t},w_{t})\big{)}\Big{]},$$

$${\tau }_t^{\star }(w_t|x_t)=$$

$$\displaystyle\frac{\exp\big{(}-\beta_{2}\sum_{u_{t}\in\mathcal{U}}\sigma_{t}^{\star}(u_{t}|x_{t})\rho_{t}(x_{t},u_{t},w_{t})\big{)}}{\sum_{w_{t}\in\mathcal{W}}\exp\big{(}-\beta_{2}\sum_{u_{t}\in\mathcal{U}}\sigma^{\star}_{t}(u_{t}|x_{t})\rho_{t}(x_{t},u_{t},w_{t})\big{)}}.$$

$${\mathcal{V}}_t^{\star }(x_t)=-\frac{1}{{\beta }_1}\sum _{u_t\in \mathcal{U}}{\sigma }_t^{\star }(u_t|x_t)\log {\sigma }_t^{\star }(u_t|x_t)$$

$$\displaystyle-\frac{1}{\beta_{2}}\log\sum_{w_{t}\in\mathcal{W}}\exp\big{(}-\beta_{2}\sum_{u_{t}\in\mathcal{U}}\sigma^{\star}_{t}(u_{t}|x_{t})\rho_{t}(x_{t},u_{t},w_{t})\big{)}.$$

$${\rho }_t(x_t,u_t,w_t)=\mathcal{R}(x_t,u_t,w_t)$$

$$+\sum _{x_{t+1}\in \mathcal{X}}\mathcal{P}(x_{t+1}|x_t,u_t,w_t){\mathcal{V}}_{t+1}^{{\sigma }_t,{\tau }_t}(x_{t+1}),$$

$${\Phi }_N(\mathbf{\sigma },\mathbf{\tau })=\sum _{x_1\in \mathcal{X}}{\mu }_1(x_1){\mathcal{V}}_1(x_1)$$

$${\mathcal{V}}_t^{\star }(x_t)=\underset{{\sigma }_t\in \Delta (U)}{\max }-\frac{1}{{\beta }_1}\sum _{u_t\in \mathcal{U}}{\sigma }_t(u_t|x_t)\log {\sigma }_t(u_t|x_t)$$

$$-\frac{1}{{\beta }_2}\log \sum _{w_t\in \mathcal{W}}\exp (-{\beta }_2\sum _{u_t\in \mathcal{U}}{\sigma }_t(u_t|x_t){\rho }_t(x_t,u_t,w_t))$$

$$\displaystyle\tau_{t}^{\star}(w_{t}|x_{t})=\arg\min_{\tau_{t}\in\Delta(W)}\frac{1}{\beta_{1}}\log\sum_{u_{t}\in\mathcal{U}}\exp\Big{(}$$

$$\displaystyle\qquad\qquad\qquad\beta_{1}\sum_{w_{t}\in\mathcal{W}}\tau_{t}(w_{t}|x_{t})\rho_{t}(x_{t},u_{t},w_{t})\Big{)},$$

$${\sigma }_t^{\star }(u_t|x_t)=$$

$$\displaystyle\frac{\exp\big{(}\beta_{1}\sum_{w_{t}\in\mathcal{W}}\tau_{t}^{\star}(u_{t}|x_{t})\rho_{t}(x_{t},u_{t},w_{t})\big{)}}{\sum_{u_{t}\in\mathcal{U}}\exp\big{(}\beta_{1}\sum_{w_{t}\in\mathcal{W}}\tau^{\star}_{t}(w_{t}|x_{t})\rho_{t}(x_{t},u_{t},w_{t})\big{)}}.$$

$$\displaystyle\mathcal{L}(\mathcal{V})(x,\sigma,\tau):=\mathbb{E}^{\sigma,\tau}\Big{[}\mathcal{R}(x,u,w)-\frac{1}{\beta_{1}}\log\sigma(u|x)$$

$$\displaystyle+\frac{1}{\beta_{2}}\log\tau(w|x)+\gamma\sum_{{x^{\prime}}\in\mathcal{X}}\mathcal{P}(x^{\prime}|x,u,w)\mathcal{V}(x^{\prime})\Big{]}.$$

$$\Psi (\mathcal{V})[x]:=\underset{\sigma \in \Delta (U)}{\max }\underset{\tau \in \Delta (W)}{\min }\mathcal{L}(\mathcal{V})(x,\sigma ,\tau ).$$

$$\Vert \Psi (\mathcal{V})-\Psi (\overline{\mathcal{V}}){\Vert }_{\infty }\le \gamma \Vert \mathcal{V}-\overline{\mathcal{V}}{\Vert }_{\infty },$$

$$\displaystyle\mathbb{E}^{\overline{\mu}_{t}}\Big{[}\mathcal{R}(x_{t},u_{t},w_{t})-\frac{1}{\beta_{1}}\log\sigma^{\star}(u_{t}|x_{t})+\frac{1}{\beta_{2}}\log\tau(w_{t}|x_{t})$$

Full text

I Introduction

A two-player zero-sum stochastic game (SG) [1] models sequential decision-making of two players with opposing objectives in a stochastic environment. An SG is played in stages. At each stage, the game is in a state, and the players choose one of their available actions simultaneously and receive payoffs. The game then transitions to a new random state according to a probability distribution which represents the stochasticity in the environment.

In an SG, each player aims to synthesize a strategy that maximizes the player’s expected payoff at the end of the game. Traditionally, a pair of optimal strategies is described by Nash equilibrium [2] according to which both players play their best-response strategies against the opponent’s strategy. The value of the game then corresponds to the expected payoff that each player receives at the end of the game, if they both play their respective equilibrium strategies.

The concept of Nash equilibrium is based on the assumptions that the players have perfect information about the environment and act rationally [3]. However, in certain scenarios, the information that a player has about the environment may not match the reality. For example, in a planning scenario, if the player obtains its information about the environment through surveillance missions performed in the past, it may face with a significantly different environment during the execution of the play. In such scenarios, playing an equilibrium strategy may dramatically decrease the player’s actual expected payoff as the strategy is computed under the assumption of perfect information.

The principle of maximum entropy prescribes a probability distribution that is \saymaximally noncommittal with regard to missing information [4]. The principle of maximum causal entropy extends the maximum entropy principle to settings where there is dynamically revealed side information that causally affects the evolution of a stochastic process [5, 6]. A distribution that maximizes the causal entropy of a stochastic process (in the absence of additional constraints) is the one that makes all admissible realizations equally probable regardless of the revealed information [7]. Therefore, the causal entropy of a player’s strategy provides a convenient way to quantify the dependence of its strategy to its level of information about the environment as well as the other player’s strategy.

In this paper, we propose a method to synthesize a pair of strategies that balances each player’s rationality with its belief about the level of missing information. Specifically, we regularize each player’s objective with the causal entropy of its strategy which is causally dependent on the history of play. Therefore, the proposed method allows the players to adjust their strategies according to different levels of misinformation by tuning a parameter that controls the importance of the regularization term. For example, in two extremes, it allows the player to be perfectly rational or to purely randomize its strategy.

We study both entropy-regularized $N$-stage and entropy-regularized discounted games, and show the existence of a value in both games. We first prove the sufficiency of Markovian and stationary strategies for both players, respectively, in $N$-stage and discounted games in order to maximize their entropy-regularized expected payoff. Then, we provide algorithms based on a sequence of convex optimization problems to compute a pair of equilibrium strategies. Finally, we demonstrate the proposed methods on a motion planning scenario, and illustrate that the introduced regularization term yields strategies that perform well in different environments.

Related work. In stochastic games literature, the idea of balancing the expected payoffs with an additional regularization term appeared recently in [8] and [9]. The work [8] proposes to bound the rationality of the players to obtain tunable behavior in video games. They study $\gamma$-discounted games and restrict their attention to stationary strategies to balance the expected payoffs with the Kullback-Leibner distance of the player’s strategies from reference strategies. In [9], authors study $N$-stage games and consider only Markovian strategies to balance the expected payoffs with the player’s sensing costs which are expressed as directed information from states to actions. Unlike this work, we introduce causal entropy of strategies as the regularization term. Additionally, we allow the player’s to follow history-dependent strategies and prove the sufficiency of Markovian strategies to attain the value in $N$-stage games.

Regularization terms are also used in matrix and extensive form games generally to learn equilibrium strategies [10], [11], [12], [13]. When each player uses the same parameter to regularize its expected payoffs with the entropy of its strategy, an equilibrium strategy profile is called a quantal response equilibrium (QRE) [3], and an equilibrium strategy of a player is referred as quantal best response [11] or logit choice strategy [10]. From a theoretical perspective, the main difference between our approach and the well-studied QRE concept [14] is that we establish the existence of equilibrium strategies even if the players use different regularization parameters. Additionally, we provide an efficient algorithm based on a convex optimization problem to compute the equilibrium strategies.

Robust stochastic games [15], [16] concern the synthesis of equilibrium strategies when the uncertainty in transition probabilities and payoff functions can be represented by structured sets. Unlike robust SG models, the proposed method in this paper can still be used when it is not possible to form a structured uncertainty set.

In reinforcement learning literature, the use of regularization terms is extensively studied to obtain robust behaviors [17], improve the convergence rates [18], and compute optimal strategies efficiently [19]. As stochastic games model multi-player interactions, our approach leverages the ideas discussed in aforementioned work to environments where an adversary aims to prevent a player to achieve its objective.

II Background

We first review some concepts from game theory and information theory that will be used in the subsequent sections.

Notation: For a sequence $x$, we write $x^{t}$ to denote $(x_{1},x_{2},\ldots,x_{t})$. Upper case symbols such as $X$ denote random variables, and lower case symbols such as $x$ denote a specific realization. The cardinality of a set $\mathcal{X}$ is denoted by $\lvert\mathcal{X}\rvert$, and the probability simplex defined over the set $\mathcal{X}$ is denoted by $\Delta(\mathcal{X})$. For $V_{1}$,V_{2}$$\in$$\mathbb{R}^{n}, we write V_{1}$$\preccurlyeq$$V_{2} to denote the coordinate-wise inequalities. We use the index set \mathbb{Z_{+}}$$=$$\{1,2,\ldots\} and the natural logarithm \log(\cdot)$$=$$\log_{e}(\cdot).

II-A Two-Player Stochastic Games

A two-player stochastic game $\Gamma$ [1] is played in stages. At each stage $t$, the game is in one of its finitely many states $\mathcal{X}$, and each player observes the current state $x_{t}$. At each state $x_{t}$, the players choose one of their finitely many actions, and the game transitions to a successor state $x_{t+1}$ according to a probability distribution \mathcal{P}$$:$$\mathcal{X}$$\times$$\mathcal{U}$$\times$$\mathcal{W}$$\rightarrow$$\Delta(\mathcal{X}) where $\mathcal{U}$ and $\mathcal{W}$ are finite action spaces for player 1 and player 2, respectively. The pair of actions, u_{t}$$\in$$\mathcal{U} and w_{t}$$\in$$\mathcal{W}, together with the current state x_{t}$$\in$$\mathcal{X} determine the payoff \mathcal{R}(x_{t},u_{t},w_{t})$$\leq$$\overline{\mathcal{R}}$$<$$\infty to be made by player 2 to player 1 at stage $t$.

A player’s strategy is a specification of a probability distribution over available actions at each stage conditional on the history of the game up to that stage. Formally, let \mathcal{H}_{t}$$=$$(\mathcal{X}$$\times$$\mathcal{U}$$\times$$\mathcal{W})^{t-1}$$\times$$\mathcal{X} be the set of all possible history of plays up to stage $t$. Then, the strategy of player 1 and player 2 are denoted by \boldsymbol{\sigma}$$=$$(\sigma_{1},\sigma_{2},\ldots) and \boldsymbol{\tau}$$=$$(\tau_{1},\tau_{2},\ldots), respectively, where $\sigma_{t}$ :$$\mathcal{H}_{t}$$\rightarrow$$\Delta(\mathcal{U}) and $\tau_{t}$ :$$\mathcal{H}_{t}$$\rightarrow$$\Delta(\mathcal{W}) for all $t$. If a player’s strategy depends only on the current state for all stages, e.g., \sigma_{t}$$:$$\mathcal{X}_{t}$$\rightarrow$$\Delta(\mathcal{U}) for all $t$, the strategy is said to be Markovian. A stationary strategy depends only on the current state and is independent of the stage number, e.g., \boldsymbol{\sigma}$$=$$(\sigma,\sigma,\ldots), where \sigma$$:$$\mathcal{X}$$\rightarrow$$\Delta(\mathcal{U}). We denote the set of all strategies, all Markovian strategies, and all stationary strategies for player i$$\in$$\{1,2\} by $\Gamma_{i}$, $\Gamma_{i}^{M}$, and $\Gamma_{i}^{S}$, respectively.

Let $\mu_{t+1}(x^{t+1},u^{t},w^{t})$ be the joint probability distribution over the history $\mathcal{H}_{t+1}$ of play which is uniquely determined by the initial state distribution $\mu_{1}(h_{1})$ through the recursive formula

[TABLE]

where h_{t}$$\in$$\mathcal{H}_{t} is the history of play up to stage $t$.

A stochastic game with the initial distribution $\mu_{1}(x_{1})$ is called an $N$-stage game, if the game ends after $N$ stages. The evaluation function for an $N$-stage game is

[TABLE]

where \overline{\mu}_{t}(\cdot)$$:=$${\mu}_{t}(\cdot)$$\sigma_{t}(\cdot)$$\tau_{t}(\cdot). Similarly, if the number of stages in the game is infinite, and the future payoffs are discounted by a factor [math]<$$\gamma$$<$$1, the game is called a $\gamma$-discounted game. The evaluation function for a $\gamma$-discounted game is

[TABLE]

The player 1’s objective is to maximize the evaluation function, i.e., its expected payoff, whereas the player 2 aims to minimize it. A stochastic game is said to have the value $\mathcal{V}^{\star}$, if for an evaluation function $f(\boldsymbol{\sigma},\boldsymbol{\tau})$, we have

[TABLE]

A pair of strategies $(\boldsymbol{\sigma}^{\star},\boldsymbol{\tau}^{\star})$ is said to be equilibrium strategies if it attains the value of the game.

It is well-known that both $N$-stage and $\gamma$-discounted games have a value for finite state and action sets [20]. Moreover, Markovian and stationary strategies are sufficient for players to attain the value in $N$-stage and $\gamma$-discounted games, respectively [1], [21].

II-B Causal Entropy

For a sequential decision-making problem where decisions depend causally on the past information such as the history of play, the causal entropy of a strategy is a measure to quantify the randomness of the strategy. Let $X^{N}$, $Y^{N}$ and $Z^{N}$ be sequences of random variables with length $N$. The entropy of the sequence $X^{N}$ causally conditioned on the sequences $Y^{N}$ and $Z^{N}$ is defined as [22]

[TABLE]

where

[TABLE]

The concept of causal entropy has recently been used to infer correlated-equilibrium strategies in Markov games [23] and to recover cost functions in inverse optimal control problems [6]. In this study, we employ causal entropy to compute an equilibrium strategy profile that balances the players’ expected payoff with the randomness of their strategies in stochastic games.

In the absence of additional constraints, a strategy \boldsymbol{\sigma}$$\in$$\Gamma_{1} that maximizes the causal entropy $H(U^{N}||X^{N},W^{N-1})$ of the player 1, which is conditioned on the revealed history of play, is the stationary strategy \boldsymbol{\sigma}$$=$$(\sigma,\sigma,\ldots) where \sigma(x)(u)$$=$$1/\lvert\mathcal{U}\rvert. Therefore, a player that maximizes the entropy of its strategy acts purely randomly regardless of the history of play. On the other hand, a player that regularizes its expected payoff with the entropy of its strategy can be thought as a player that balances its rationality with its belief about the correctness of the underlying transition model of the environment.

III Problem Statement

We first consider entropy-regularized $N$-stage games for which we define the evaluation function as

[TABLE]

where \beta_{1},\beta_{2}$$>[math] are regularization parameters that adjust for players the importance of the randomness in their strategies. Note that, when \beta_{1}$$=$$\beta_{2}$$=$$\infty, both players act perfectly rational, and we recover the evaluation function (II-A). Additionally, since the play is simultaneous, the information of a player’s strategy at a given stage is not revealed to the other player. Hence, at each stage, players are allowed to condition their strategies only to observed history of play.

Problem 1: Provide an algorithm to synthesize, if exists, equilibrium strategies in entropy-regularized $N$-stage games.

We next consider stochastic games that are played in infinite stages, and introduce entropy-regularized $\gamma$-discounted games for which we define the evaluation function as

[TABLE]

where H_{t}$$=$$(X^{t},U^{t-1},W^{t-1}), i.e., the admissible histories of play at stage $t$. Note that in the evaluation function (III), we discount players’ future entropy gains as well as the expected payoff in order to ensure the finiteness of the evaluation function.

Problem 2: Provide an algorithm to synthesize, if exists, equilibrium strategies in entropy-regularized $\gamma$-discounted games.

IV Existence of Values and The Computation of Optimal Strategies

In this section, we analyze entropy regularized $N$-stage and $\gamma$-discounted games, and show that both games have values. Then, we provide algorithms to synthesize equilibrium strategies that attain the corresponding game values.

IV-A Entropy-Regularized $N$-Stage Games

Searching optimal strategies that solve a stochastic game with the evaluation function $\Phi_{N}(\boldsymbol{\sigma},\boldsymbol{\tau})$ in the space of all strategies can be intractable for large $N$. We begin with establishing the existence of optimal strategies for both players in the space of Markovian strategies.

Proposition 1

Markovian strategies are sufficient for both players to attain, if exists, the value in entropy-regularized N-stage games, i.e.,

[TABLE]

Proof: See Appendix A. $\quad\Box$

Next, we show that entropy-regularized $N$-stage games have a value. Let \rho_{t}$$:$$\mathcal{X}$$\times$$\mathcal{U}$$\times$$\mathcal{W}$$\rightarrow$$\mathbb{R} be a function and $x_{t}$ be a fixed state. Additionally, let

[TABLE]

be the evaluation function for a \sayone-shot game in which the game starts from the state $x_{t}$ and ends after both players play their one-step strategy.

Proposition 2

A stochastic game with the evaluation function (IV-A) has a value, i.e.,

[TABLE]

Proof: It is clear that $\mathcal{V}_{t}(x_{t})$ is a continuous function that is concave in $\sigma_{t}$ and convex in $\tau_{t}$. Additionally, $\Delta(\mathcal{U})$ and $\Delta(\mathcal{W})$ are compact convex sets. The result follows from von Neumann’s minimax theorem [24].$\quad\Box$

The following proposition states that one can compute the value of the one shot game (IV-A) and synthesize equilibrium strategies by solving a convex optimization problem.

Proposition 3

For a given one-shot game with the evaluation function (IV-A), optimal strategies $(\sigma_{t}^{\star},\tau_{t}^{\star})$ satisfy

[TABLE]

Furthermore, the unique value $\mathcal{V}^{\star}_{t}(x_{t})$ of the game is given by

[TABLE]

Proof: See Appendix A.$\quad\Box$

It is worth noting that the objective function of the optimization problem given in (3) is strictly concave, and therefore, optimal strategies $\sigma_{t}^{\star}(u_{t}|x_{t})$ and $\tau_{t}^{\star}(w_{t}|x_{t})$ are unique. Additionaly, an optimal strategy with the form (10) is known in the economics literature as quantal best response [14], and for \beta_{1}$$=$$\beta_{2}$$<$$\infty, the optimal strategies form the well-studied quantal response equilibrium strategies [3].

We remark that the optimization problem in (3) has a closed-form solution which is a function of the optimal strategy $\tau_{t}^{\star}(w_{t}|x_{t})$. However, since the closed-form expressions for equilibrium strategies constitute a system of coupled nonlinear equations, the convex optimization formulation provides a more convenient way to compute equilibrium strategies.

Utilizing the results of above propositions, we now reformulate an entropy-regularized $N$-stage game as a series of \sayone-shot games through the use of Bellman recursions. Let

[TABLE]

for t$$=$$1,\ldots,N where \mathcal{V}_{N+1}^{\sigma_{t},\tau_{t}}(x_{N+1})$$=$$\mathcal{R}(X_{N+1}). Then, it can be easily verified that

[TABLE]

for a given initial distribution $\mu_{1}(x_{1})$. Consequently, we obtain the following result.

Theorem 1

Entropy-regularized $N$-stage games have a value.

Proof: Due to Proposition 1, we can focus on Markovian strategies to find an equilibrium point in $N$-stage games. We start from the stage k$$=$$N and compute the value of the one shot game (IV-A), which exists due to Proposition 2. Using (IV-A) with (IV-A) for k$$=$$N$$-$$1,N$$-$$2,\ldots,1, we compute the value of N$$-$$k$$+$$1 stage games. As a result, the claim follows due to the equivalence given in (13).$\quad\Box$

Algorithm 1 summarizes the computation of the pair $(\boldsymbol{\sigma}^{\star},\boldsymbol{\tau}^{\star})$ of optimal strategies for entropy-regularized $N$-stage games.

Remark: In certain scenarios, one of the players may prefer to play perfectly rationally against a boundedly rational opponent, e.g., \beta_{2}$$=$$\infty. In that case, it is still possible to compute equilibrium strategies by solving a convex optimization problem at each stage. The value of the one-shot game (IV-A) still exists due to the arguments provided in the proof of Proposition 2. However, the form of optimal strategies slightly changes to

[TABLE]

It is important to note that, if \beta_{i}$$=$$\infty for some i$$\in$$\{1,2\}, equilibrium strategies may be not unique since the function $\log\sum\exp(\cdot)$ is not strictly convex over its domain [25].

IV-B Entropy-Regularized $\gamma$-Discounted Games

In this section, we focus on Markovian strategies, whose optimality for $N$-stage games is shown in Proposition 1. Let \mathcal{V}$$\in$$\mathbb{R}^{\lvert\mathcal{X}\rvert} be a real-valued function, and for a given x$$\in$$\mathcal{X}, $\mathcal{L}(\mathcal{V})(x,\cdot,\cdot)$ $:$ \mathcal{U}$$\times$$\mathcal{W}$$\rightarrow$$\mathbb{R} be a function such that

[TABLE]

As discussed in Proposition 2, a one-shot game with the evaluation function $\mathcal{L}(\mathcal{V})(x,\sigma,\tau)$ has a value. Therefore, we can introduce the Shapley operator \Psi$$:$$\mathcal{V}$$\rightarrow$$\Psi(\mathcal{V}) from $\mathbb{R}^{\lvert\mathcal{X}\rvert}$ to itself specified, for all x$$\in$$\mathcal{X}, as

[TABLE]

It is clear that the operator $\Psi$ satisfies two key properties: monotonicity, i.e., \mathcal{V}$$\preccurlyeq$$\overline{\mathcal{V}} implies \Psi(\mathcal{V})$$\preccurlyeq$$\Psi(\overline{\mathcal{V}}), and reduction of constants, i.e., for any k$$\geq[math], \Psi(\mathcal{V}$$+$$k\boldsymbol{1})[x]$$=$$\Psi(\mathcal{V})[x]$$+$$\gamma k for all x$$\in$$\mathcal{X}. Consequently, it is straightforward to show that the operator $\Psi$ is a contraction mapping [26]. Specifically, we have

[TABLE]

where \lVert\mathcal{V}\rVert_{\infty}$$=$$\max_{x\in\mathcal{X}}\mathcal{V}(x). We omit the details since similar results can be easily found in the literature [27]. Then, by Banach’s fixed-point theorem [28], we conclude that the operator $\Psi$ has a unique fixed point which satisfies \mathcal{V}^{\star}$$=$$\Psi\mathcal{V}^{\star}.

Next, we need to show that the fixed point $\mathcal{V}^{\star}$ is indeed the value of the entropy-regularized $\gamma$-discounted game. Let \boldsymbol{\sigma}^{\star}$$=$$(\sigma^{\star},\sigma^{\star},\ldots) be a stationary strategy such that $\sigma^{\star}$ is a one-step strategy for player 1 satisfying the fixed point equation, and $\boldsymbol{\tau}$ be an arbitrary Markovian strategy for player 2. Denoting by $h_{t}$ the history of play of length $t$, one has, by definition of $\Psi$ and $\boldsymbol{\sigma}^{\star}$,

[TABLE]

This expression can further be written as

[TABLE]

Multiplying by $\gamma^{t-1}$, taking expectations and summing over 1$$\leq$$t$$<$$k, one obtains

[TABLE]

Taking the limit as k$$\rightarrow$$\infty and using Proposition 1, we obtain

[TABLE]

Similarly, when player 2 plays the optimal stationary strategy \boldsymbol{\tau}^{\star}$$=$$(\tau^{\star},\tau^{\star},\ldots) against an arbitrary Markovian strategy $\boldsymbol{\sigma}$ of player 1, we have

[TABLE]

Then, the combination of (17) and (18) implies the following result.

Theorem 2

Entropy-regularized $\gamma$-discounted games have a value which satisfies \Psi(\mathcal{V})$$=$$\mathcal{V}. Furthermore, stationary strategies are sufficient for both players to attain the game value, i.e.,

[TABLE]

Computation of optimal strategies is just an extension of Algorithm 1. Note that for $\gamma$-discounted games, we use the same one-shot game introduced in (IV-A). Therefore, optimal decision rules at each stage has the form (3) and (10). Consequently, to compute the optimal strategies, we initialize Algorithm 1 with an arbitrary value vector \mathcal{V}$$\in$$\mathbb{R}^{\lvert\mathcal{X}\rvert} and iterate until convergence, which is guaranteed by the existence of a unique fixed point.

V A Numerical Example

In this section, we demonstrate the proposed strategy synthesis method on a motion planning scenario that we model as an entropy-regularized $\gamma$-discounted game. To solve the convex optimization problems required for the computation of equilibrium strategies, we use ECOS solver [29] through the interface of CVXPY [30]. All computations are performed by setting \gamma$$=$$0.8.

As the environment model, we consider a 5$$\times$$5 grid world which is given in Figure 1 (top left). The brown grid denotes the initial position of the player 1 which aims to reach the goal (green) state. The red grid is the initial position of the player 2 whose aim is to catch the player 1 before reaching the goal state. Finally, black grids represent walls.

Let x$$=$$(s_{1},s_{2}) be the current state of the game such that x[1]$$=$$s_{1} and x[2]$$=$$s_{2} are the positions of the player 1 and the player 2, respectively. At each state, the action space for both players is given as \mathcal{U}$$=$$\mathcal{W}$$=$$\{right,left,up,down,stay\}. For simplicity, we assume deterministic transitions, i.e., \mathcal{P}(x,u,w)$$\in$$\{0,1\} for all x$$\in$$\mathcal{X}, u$$\in$$\mathcal{U} and w$$\in$$\mathcal{W}. If a player takes an action for which the successor state is a wall, the player stays in the same state with probability 1.

For a given (x,u,w)$$\in$$\mathcal{X}$$\times$$\mathcal{U}$$\times$$\mathcal{W}, we encode the payoff function $\mathcal{R}(x,u,w)$ as the sum of two functions such that \mathcal{R}(x,u,w)$$=$$\mathcal{R}_{1}(x,u,w)$$+$$\mathcal{R}_{2}(x,u,w) where

[TABLE]

Note that the payoff function defines a zero-sum game which is won by the player 1, if it reaches the goal state before getting caught, and by the player 2, if it catches the player 1 before reaching the goal state.

We first compute Nash equilibrium strategies in the absence of causal entropy terms, i.e., \beta_{1}$$=$$\beta_{2}$$=$$\infty, by employing standard linear programming formulation [31] for zero-sum games. Starting from the initial state, an equilibrium strategy for the player 1 is to move towards the goal state by taking the action $right$ deterministically, and for the player 2 is to chase the player 1 by taking the action $up$ in the first two stages, and then, take the action $right$ until reaching the goal state. Therefore, a perfectly rational player 1 wins the game with probability 1 no matter what strategy is followed by the player 2.

To illustrate the drawback of playing with perfect rationality, we assume that there is another wall in the environment about which the players have no information while they compute the equilibrium strategies, i.e., the players use the nominal environment (top left) to compute the equilibrium strategies. First, we consider the case that the wall is between the goal state and the player 1, as shown in Figure 1 (middle left). In this case, if the player 1 follows the Nash equilibrium strategy, the probability that it reaches the goal state becomes zero. Therefore, following the Nash equilibrium strategy makes the player 1 significantly vulnerable to such changes in the environment.

To investigate the tradeoff between rationality and randomness, we compute 9 different strategies for player 1 by using \beta_{1}$$=$$\beta_{2}$$=$$2,3,\ldots,10, and let it play against the perfectly rational player 2 which follows its Nash equilibrium strategy computed on the nominal environment (top left). The winning probabilities of player 1 under different strategies are shown in Figure 1 (right) for the corresponding environments given in Figure 1 (left). This specific scenario demonstrates that, by choosing \beta$$=$$6, the player 1 can obtain a robust behavior against unforeseen changes in the environment, i.e., the winning probability is around 20%, without sacrificing too much from its optimal performance, i.e., around 15%, if the structure of the environment remains the same. It is worth noting that the asymptotical performance of the player 1 as \beta$$\rightarrow$$\infty approaches to its performance under Nash equilibrium strategy as discussed in [14]. Additionally, the importance of the randomness for the player 1 increases as \beta$$\rightarrow[math], and using smaller $\beta$ values negatively affects the performance after a critical point, i.e., \beta$$=$$4.

Finally, one can argue that the tradeoff occurs in this specific scenario only if the unpredicted wall is between the player 1 and the goal state. To justify the choice of $\beta$ value, we also consider the scenario in which the unexpected wall occupies another state which is shown in Figure 1 (bottom left). In this case, as shown in Figure 1 (bottom right), the use of \beta$$=$$6 result in a strategy that guarantees around 80% winning probability. Therefore, the entropy-regularized strategy of the player 1 still provides an advantage against unpredicted changes without sacrificing too much from the optimal performance.

VI Conclusions and Future Work

We consider the problem of two-player zero-sum stochastic games with entropy regularization, wherein the players aim to maximize the causal entropy of their strategies in addition to the conventional expected payoff. We show that equilibrium strategies exist for both entropy-regularized $N$-stage and entropy-regularized $\gamma$-discounted games, and can be computed by solving a convex optimization problem. In numerical examples, we applied the proposed approach to a motion planning scenario and observed that by tuning the regularization parameter, a player can synthesize robust strategies that perform well in different environments against a perfectly rational opponent.

Extending this work to multi-agent reinforcement learning settings, as discussed in [8], is an interesting future direction. Future work can also investigate the effect of entropy regularization term on the convergence rate of learning algorithms as discussed in [18].

Appendix A

Proof of Proposition 1: We show the sufficiency of Markovian strategies only for the maximin problem. The proof for the minimax formulation follows the same lines with the arguments provided below.

The proof is based on backward induction on the stage number 1$$\leq$$k$$\leq$$N. Let

[TABLE]

be the value of the N$$-$$k stage problem. Then, we can write the value of N$$-$$k stage problem recursively as

[TABLE]

Base step: k$$=$$N. Let $\sigma_{N}$ and $\tau_{N}^{{\star}}$ be an arbitrary strategy for player 1 and the optimal strategy for player 2 at stage $N$, respectively. Let

[TABLE]

be the joint distribution induced by $\sigma_{N}(u_{N}|h_{N})$ and $\tau_{N}^{\star}(w_{N}|h_{N})$. Additionally, let $\lambda_{N}(x_{N},w_{N})$ and $\lambda_{N}(x_{N})$ be the marginal distributions of $\lambda_{N}(h_{N},u_{N},w_{N})$. We construct a new strategy for player 2 as \overline{\tau}_{N}(w_{N}|x_{N})$$:=$$\frac{\lambda_{N}(x_{N},w_{N})}{\lambda_{N}(x_{N})}. Let

[TABLE]

be the joint distribution induced by $\overline{\tau}_{N}(w_{N}|x_{N})$. Then, by construction, we have \overline{\lambda}_{N}(x_{N},u_{N},w_{N})$$=$${\lambda}_{N}(x_{N},u_{N},w_{N}), which can be easily verified by calculating the corresponding marginal distributions. (For a similar strategy construction, see Theorem 5.5.1 in [28].)

The inner optimization problem in (Appendix A) for k$$=$$N reads

[TABLE]

where

[TABLE]

Since the strategy $\sigma_{N}$ is arbitrarily chosen, it is sufficient to show that J_{N}^{c}(\lambda_{N})$$=$$J_{N}^{c}(\overline{\lambda}_{N}) and J_{N}^{H}(\lambda_{N})$$\leq$$J_{N}^{H}(\overline{\lambda}_{N}) in order to establish the sufficiency of Markovian strategies for player 2. The first equality holds by construction. (Note that the $\log(\cdot)$ term is indifferent to changes in the strategy of player 2.) The second inequality can be derived as

[TABLE]

where (21) holds since conditioning reduces entropy [32], (22) is because \lambda_{N}$$=$$\overline{\lambda}_{N} by construction, and (23) is due to the fact that $\overline{\tau}_{N}(w_{N}|x_{N})$ is a Markovian strategy. Consequently, for any strategy chosen by player 1 in stage $N$, player 2 has a best response strategy in the space of Markovian strategies.

Next, we can assume that player 2 uses a Markovian strategy and show through a similar strategy construction explained above that player 1 has an optimal strategy in the space of Markovian strategies. As a result, the value $\mathcal{V}_{N}$ depends on the joint distribution $\mu_{N}(x^{N},u^{N-1},w^{N-1})$ only through its marginal $\mu_{N}(x_{N})$ and becomes a function of the marginal $\mu_{N}(x_{N})$ only.

Inductive step: k$$=$$t Assume that Markovian strategies suffice for both players for k$$=$$t$$+$$1,t$$+$$2,\ldots,N. Then, by induction hypothesis, $\mathcal{V}_{t+1}$ is a function of $\mu_{t+1}(x_{t+1})$ only. Therefore, using the similar construction to the case k$$=$$N, we can construct Markovian strategies $\overline{\sigma}_{t}(u_{t}|x_{t})$ and $\overline{\tau}_{t}(w_{t}|x_{t})$ such that the objective function in the right hand side of (Appendix A) attained by $\overline{\sigma}_{t}$ and $\overline{\tau}_{t}$ is equal to the value of $N-t$ stage problem. As a result, we conclude that Markovian strategies are sufficient for both players to solve the maximin problem (III). $\quad\Box$

Proof of Proposition 3: Since the one-shot game has a value, without loss of generality, we focus on the problem $\max_{\sigma_{t}\in\Delta(\mathcal{U})}\min_{\tau_{t}\in\Delta(\mathcal{W})}\mathcal{V}_{t}(x_{t})$. For notational convenience, we rewrite the problem as

[TABLE]

where Q^{ij}$$=$$\sigma_{t}(u_{t}|x_{t}), Q^{ik}$$=$$\tau_{t}(w_{t}|x_{t}) and \rho_{ijk}$$=$$\rho_{t}(x_{t},u_{t},w_{t}). Note that due to constraints \sum_{j}Q^{ij}$$=$$1 and \sum_{k}Q^{ik}$$=$$1, we can replace $\frac{1}{\beta_{1}}\sum_{jk}Q^{ik}Q^{ij}\log Q^{ij}$ and $\frac{1}{\beta_{2}}\sum_{jk}Q^{ij}Q^{ik}\log Q^{ik}$ by $\frac{1}{\beta_{1}}\sum_{j}Q^{ij}\log Q^{ij}$ and $\frac{1}{\beta_{2}}\sum_{k}Q^{ik}\log Q^{ik}$, respectively. For now, we neglect the non-negativity constraints and write the Lagrangian for the above optimization problem as

[TABLE]

where $\lambda^{j}$,$\lambda^{k}$ are Lagrange multipliers. Then, taking derivative with respect to $Q^{ik}$ and equating it to zero, we obtain

[TABLE]

Rearranging terms and using the constraint $\sum_{k}Q^{ik}=1$, we obtain

[TABLE]

which is the same as (10). Note that the resulting strategy also satisfies the non-negativity constraint. Plugging $Q^{ik}_{\star}$ into Lagrangian $L$, we obtain the optimization problem given in (3) for which the optimal variables correspond to the optimal strategy of player 1. Similarly, the optimal value (3) of the resulting optimization problem is the value of the game. Uniqueness of the value follows from the fact that the value of the game is the optimal value of a convex optimization problem given in (3). $\quad\Box$