Worst-Case Analysis for a Leader-follower Partially Observable Stochastic Game

TL;DR

This paper introduces a worst-case analysis framework for leader-follower partially observable stochastic games, providing a method to compute lower bounds on the leader's value function under uncertainty and limited knowledge.

Contribution

It develops a novel worst-case analysis approach for leader-follower stochastic games with partial observability, including a computational method for lower bound estimation.

Findings

Provides a new solution procedure for the leader's value function.

Demonstrates the approach on a liquid egg production security problem.

Offers insights into decision-making under uncertainty with limited information.

Abstract

Partially observable stochastic games provide a rich mathematical paradigm for modeling multi-agent dynamic decision making under uncertainty and partial information. However, they generally do not admit closed-form solutions and are notoriously difficult to solve. Also, in reality, each agent often does not have complete knowledge of the other agent. This paper studies a leader-follower partially observable stochastic game where the leader has little knowledge of the adversarial follower's reward structure, level of rationality, and process for gathering and transmitting data relevant for decision making. We introduce the worst-case analysis to the partially observable stochastic game to cope with this lack of knowledge and determine the best worst-case value function of the leader. The resulting problem from the leader's perspective has a simple sufficient statistic; however, different from a classical partially observable Markov decision process, the value function of the resulting problem may not be convex. We design a viable and computationally attractive solution procedure for computing a lower bound of the leader's value function as well as its associated control policy in the finite planning horizon. We illustrate the use of the proposed approach in a liquid egg production security problem.

Tables

Table 1. Table 1: Γ T ​ ( s L ) subscript Γ 𝑇 superscript 𝑠 𝐿 \Gamma_{T}(s^{L}) where s L ≠ superscript 𝑠 𝐿 absent s^{L}\neq “Attacked”, in terms of the number of qualified products.

${\gamma }^{1,1}$	${\gamma }^{1,2}$	${\gamma }^{2,1}$	${\gamma }^{2,2}$	${\gamma }^{3,1}$
916	906	0	756	0
723	906	916	756	703
746	906	786	756	726
-100	-100	-100	-100	-100

Equations

$$v_0^{L,{\pi }^{L,\ast },{\pi }^{F,\ast }}({\zeta }_0^L)=\underset{{\pi }^L\in {\Pi }^L}{\max }\underset{{\pi }^F\in {\Pi }^F}{\min }E\{\sum _{t=0}^T{\beta }^t{r}^L(s_t,a_t)|{\zeta }_0^L\},$$

$$v_t^L({\zeta }_t^L)$$

$$\displaystyle+\beta\sum_{z^{L}_{t+1}}\sum_{s^{L}_{t+1}}P(z^{L}_{t+1},s^{L}_{t+1}|\zeta^{L}_{t},a_{t})v^{L}_{t+1}(\zeta^{L}_{t+1})\Bigg{\}}.$$

$$x_{t+1}^L={\lambda }^L(z_{t+1}^L,s_{t+1}^L,s_t^L,x_t^L,a_t)$$

$${\lambda }^L(z_{t+1}^L,s_{t+1}^L,s_t^L,x_t^L,a_t)=P(s_{t+1}^F|{\zeta }_{t+1}^L)=P(s_{t+1}^F|z_{t+1}^L,s_{t+1}^L,a_t,{\zeta }_t^L)$$

$$=\frac{P(z_{t+1}^L,s_{t+1}|a_t,{\zeta }_t^L)}{P(z_{t+1}^L,s_{t+1}^L|a_t,{\zeta }_t^L)}=\frac{{\sum }_{s_t^F}P(z_{t+1}^L,s_{t+1}|s_t,a_t)x^L(s_t^F)}{{\sum }_{s_{t+1}^F}{\sum }_{s_t^F}P(z_{t+1}^L,s_{t+1}|s_t,a_t)x^L(s_t^F)},$$

$${\sigma }^L(z_{t+1}^L,s_{t+1}^L,s_t^L,x_t^L,a_t)=P(z_{t+1}^L,s_{t+1}^L|a_t,{\zeta }_t^L),$$

$$\displaystyle[Hv](s^{L},x^{L})=\max_{a^{L}\in A^{L}}\min_{a^{F}\in A^{F}}\Bigg{\{}x^{L}r^{L}(s^{L},a)$$

$$\displaystyle+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)v(s^{L^{\prime}},\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a))\Bigg{\}},$$

$$v_t^L({\zeta }_t^L)$$

$$v(s^{L^{\prime }},{\lambda }^L(z^{L^{\prime }},s^{L^{\prime }},s^L,x^L,a))={\lambda }^L(z^{L^{\prime }},s^{L^{\prime }},s^L,x^L,a){\gamma }^{l(z^{L^{\prime }},s^{L^{\prime }},s^L,x^L,a)},\gamma \in \Gamma (s^{L^{\prime }}),$$

$${\gamma }^{{}^{\prime }a}(s^F)$$

$$[Hv](s^L,x^L)=\underset{a^L}{\max }\underset{a^F}{\min }\left\{x^L{\gamma }^{{}^{\prime }a}:{\gamma }^{{}^{\prime }a}\in {\Gamma }^{\prime }(s^L)\right\},$$

$$\displaystyle 0\leq[Hu](s^{L},x^{L})-[Hv](s^{L},x^{L})\leq\max_{a^{L}}\Bigg{\{}\min_{a^{F}\in A^{F}}\bigg{[}x^{L}r^{L}(s^{L},a)$$

$$\displaystyle+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)u(s^{L^{\prime}},\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a))\bigg{]}$$

$$\displaystyle-\min_{a^{F}\in A^{F}}\bigg{[}x^{L}r^{L}(s^{L},a)+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)v(s^{L^{\prime}},\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a))\bigg{]}\Bigg{\}}$$

$$\displaystyle\min_{a^{F}\in A^{F}}\bigg{[}x^{L}r^{L}(s^{L},a)+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)u(s^{L^{\prime}},\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a))\bigg{]}$$

$$\displaystyle-\min_{a^{F}\in A^{F}}\bigg{[}x^{L}r^{L}(s^{L},a)+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)v(s^{L^{\prime}},\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a))\bigg{]}$$

$$\displaystyle\leq\bigg{[}x^{L}r^{L}(s^{L},a^{L},a^{F*})+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a^{L},a^{F*})u(s^{L^{\prime}},\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a^{L},a^{F*}))\bigg{]}$$

$$\displaystyle-\bigg{[}x^{L}r^{L}(s^{L},a^{L},a^{F*})+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a^{L},a^{F*})v(s^{L^{\prime}},\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a^{L},a^{F*}))\bigg{]}$$

$$\le \beta ||u-v||,\forall a^L$$

$$\displaystyle[H^{a^{L}}v](s^{L},x^{L})=\min_{a^{F}\in A^{F}}\Bigg{\{}x^{L}r^{L}(s^{L},a)$$

$$\displaystyle+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)v(s^{L^{\prime}},\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a))\Bigg{\}},$$

$$Hv=\underset{a^L\in A^L}{\max }[H^{a^L}v].$$

$$\displaystyle[H^{a^{L}}\tilde{v}^{L}](s^{L},x^{L})=\min_{a^{F}\in A^{F}}\Bigg{\{}x^{L}r^{L}(s^{L},a)$$

$$\displaystyle+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)\times\min[\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)\gamma^{\prime}:\gamma^{\prime}\in\tilde{\Gamma}(s^{L^{\prime}})]\Bigg{\}}$$

$$\displaystyle=\min_{a^{F}\in A^{F}}\Bigg{\{}x^{L}r^{L}(s^{L},a)+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\min[\sum_{s^{F^{\prime}}}\sum_{s^{F}}P(z^{L^{\prime}},s^{\prime}|s,a)x^{L}(s^{F})\gamma^{\prime}(s^{F^{\prime}}):\gamma^{\prime}\in\tilde{\Gamma}(s^{L^{\prime}})]\Bigg{\}}$$

$$\displaystyle=\min\Bigg{\{}x^{L}\gamma:\gamma\in G(s^{L},a^{L})\Bigg{\}},$$

$$[H{\tilde{v}}^L](s^L,x^L)=\underset{k_1}{\max }\underset{k_2}{\min }\{x^L{\gamma }^{k_1,k_2}:{\gamma }^{k_1,k_2}\in \Gamma (s^L)\}.$$

$$\displaystyle G(s^{L},a^{L})=\cup_{a^{F}\in A^{F}}\Bigg{\{}r^{L}(s,a)+\beta\sum_{z^{L^{\prime}}}\sum_{s^{{}^{\prime}}}P(z^{L^{\prime}},s^{\prime}|s,a)\gamma^{\prime}(s^{F^{\prime}}):\gamma^{\prime}\in\tilde{\Gamma}_{t+1}(s^{L^{\prime}})\Bigg{\}}.$$

$${\bar{v}}^{k_1}(s^L,x^{\prime })$$

$${\bar{v}}^{a^L}(s^L,x^{\prime })$$

$$u\equiv \min$$

$$s.t.$$

$$-z_1\le -x{\gamma }^{k_1,k_2}+M(1-{\rho }^{k_1,k_2}),{\gamma }^{k_1,k_2}\in {\Gamma }_t^c(s^L);$$

$$\sum _{k_2=1}^{K_2(s^L,k_1)}{\rho }^{k_1,k_2}=1,\forall k_1=1,...,K_1(s^L);$$

$${\rho }^{k_1,k_2}\in \{0,1\},x\in X^L,z_1,z_2\in R,$$

$$d(v^1,v^2)(s^L)=\underset{x^L\in X^L}{\max }|v^1(s^L,x^L)-v^2(s^L,x^L)|.$$

$$min$$

Full text

Keywords: worst-case analysis; partially observable Markov decision process; partially observable stochastic game.

1 Introduction

Stochastic games are classical dynamic game models, where the state of the system evolves on the basis of the current state and actions taken by all agents. However, these models do not consider the fact that in reality, an agent in a multi-agent scenario is likely to have only partial and noise-corrupted data about other agents. Partially observable stochastic games (POSGs) generalize stochastic games by taking into consideration this fact, and provide a rich normative framework for multiple intelligent agents with distinct objectives to dynamically control the operation of a system under uncertainty and partial information. Unfortunately, while game theorists have heavily studied Bayesian games and stochastic games, the literature on POSGs is relatively sparse. POSGs generally do not admit closed-form solutions and suffer from significant computational challenges. Tractable algorithms for computing control policies for general-sum POSGs are still rare, and both the Artificial Intelligence (AI) and Operations Research (OR) communities have focused on POSGs with special structures including zero-sum POSGs (Ghosh et al. 2004; Saha 2014) and common-payoff POSGs (Seuken and Zilberstein 2005; Oliehoek and Amato 2016) over the past decades.

A leader-follower POSG introduced in Chang et al. (2015a, 2015b) is a new general-sum POSG, where each agent knows its own state but only has possibly inaccurate/incomplete observation of the other agent’s state. At the beginning of the game, the leader selects its policy first, and then the follower determines its best response policy, with the complete knowledge of the leader’s policy (e.g., adversaries spend time in learning defender’s policies (Information Operations, 2014)). At each stage, the selected policy pair determines actions for each agent simultaneously, and then the system transitions to a new state and each agent receives a new observation. Each agent’s policy selects action to achieve its objective, based on the entire history of its own current and past observations, states, and actions.

The (single-period) leader-follower game, also called the Stackelberg game, has wide applications and has been successfully implemented in many real security problems. A main focus of these applications is to provide decision support to the defender (commonly modeled as the leader) who is protecting a set of critical nodes against adversaries. Such examples include the placement of checkpoints and canine units at Los Angeles International Airport (Jain et al. 2010) and the scheduling of patrols in the Port of Boston (An et al. 2014). However, in the presence of intelligent agents who can respond and adjust their actions in ever-changing environments over time, many of these problems also have a dynamic nature that cannot be fully addressed by single-period games (e.g., an adversary may choose another target if the current target is well protected). The leader-follower POSG is able to explicitly model the dynamic interaction between agents and determine dynamic defense policies for the leader that promptly adjust defensive resource allocation over time for protection, given all available real-time data. This development is consistent with the emerging “Moving Target Defense” belief that nowadays it is impossible to attain perfect security and the aim should be to develop dynamic systems that are defensible rather than perfectly secure (Department of Homeland Security 2015).

Game-theoretic approaches commonly assume that each agent has complete knowledge of both agents’ reward structures, dynamics, and data sensing and transmission systems and that each agent’s objective is to maximize its expected reward criterion. However, these assumptions may not be realistic in many scenarios (Camerer 2011). The intent of an adversary can span a wide range of possibly unknown issues (Bier et al. 2007), perfect rationality is often an unlikely human behavior (March 1978), and action selection may be affected by a variety of issues, such as task complexity, the interplay between emotion and cognition, etc. (Conlisk 1996).

In this paper, we consider a leader-follower general-sum POSG where: (i) the objective of the adversarial follower is unknown to the leader; (ii) at each decision epoch, it is unclear what information (or observations) that the follower has collected and how the follower will make its decision; (iii) the follower can be irrational; and (iv) the state of each agent cannot be precisely observed by the other agent. This is realistic in many situations, for example, when the leader is facing with a new unknown adversary. The intent of this research is to determine the best worst-case value function and the corresponding control strategy for the leader under these circumstances.

The worst-case analysis is a popular approach in single-period games to cope with the uncertainty of an agent’s behavior towards others. For example, Gilboa and Schmeidler (1989), Lo(1996), and Marinacci(2000) examined normal form games where an agent’s action is not exactly known by the other agent. Aghassi and Bertsimas(2006) and Yolmeh and Baykal-Gursoy(2017) used this approach to contend with the payoff uncertainty. Simchi-Levi and Wei(2015) and Caprara et al.(2016) also determined performance benchmarks in one-shot security planning via the same approach. The benchmarks from the worst-case analysis were further used to reveal the value of improved understanding of the behavior of adversaries in single-period security applications (Nguyen et al. 2013). Kardes et al.(2011) introduced robust optimization to stochastic games where reward structure and/or transition probabilities are uncertain. However, the related analysis has not been examined for general-sum POSGs. This research introduces the worst-case analysis to the leader-follower POSG to reduce this gap in the literature.

Contributions of this paper are summarized as follows.

1. (i)

We introduce the worst-case analysis to a leader-follower POSG to consider the case where the leader has little knowledge of the adversarial follower. This is a first step for further evaluating the value of an improved understanding of the adversarial follower in a dynamic, multi-agent partially observable stochastic system. 2. (ii)

(Theoretically) We show that the POSG under the worst-case analysis is a single-agent dynamic decision-making problem based solely on information of the leader. This model is unique from the zero-sum POSG and has a sufficient statistic more computationally tractable than the existing sufficient statistics presented in the POSG literature. We investigate the structural properties of the leader’s optimal value function and show that it may not be convex. 3. (iii)

(Computationally) The non-convex structural result limits the usefulness of existing partially observable Markov decision process (POMDP) algorithms in our problem. While the worst-case model can also be viewed as a POMDP with imprecise parameters under the criterion of “maxmin”, currently there are no general algorithms for these problems. In order to establish a bottom-line performance for the leader, we develop a novel backward recursive algorithm to construct a lower bound for the leader’s finite-horizon value function and to determine its associated policy. We evaluate the quality of the solution and show that the lower bound is no worse than the value function associated with the second best leader’s action. We also show that this algorithm can approximately determine a lower bound for the infinite planning horizon problem. 4. (iv)

(Application wise) We illustrate the use of the proposed model and solution procedure via a security problem, where the operations manager of a liquid egg production plant is protecting the facility against an adversary who intends to insert a biological toxin into the system. We test and validate the effectiveness of the developed dynamic defensive strategy using simulation.

This paper is organized as follows. Section 2 presents a literature review on stochastic games with incomplete information, POSGs, and POMDPs with imprecise parameters. In Section 3, we introduce the worst-case analysis model for the general-sum leader-follower POSG, where the objective of the leader is to maximize the expected total discounted reward under the worst-case scenario of the follower. Section 4 presents the structural results of the leader’s optimal value function, followed by their computational implications. In Section 5, we propose a three-step solution procedure for constructing a lower bound of the leader’s value function and its policy, consisting of the PURGE-step, the DOMINANCE-step, and the APPROXIMATION-step. We discuss each of these steps in detail in Sections 6-8. Specifically, Section 6 utilizes an POMDP algorithm to eliminate redundant vectors in constructing the value function for a given leader action; Section 7 presents a geometric approach and a mixed integer program to determine the optimal value function; and Section 8 approximates the resulting value function by a piecewise linear and concave function. The approximation solution is used in the next iteration of the recursive algorithm. We also analyze the error bound of this approach and show our algorithm can be used to construct a lower bound for the leader’s infinite-horizon optimal value function. Section 9 illustrates the use of the developed approach in a security application. Finally, Section 10 summarizes research results and discusses future research directions.

2 Literature Review

This section briefly reviews stochastic games with incomplete information, partially observable stochastic games, and the POMDP with imprecise parameters.

2.1 Stochastic Games with Incomplete Information

Stochastic games were first developed in Shapley(1953). Afterwards, many extensions were developed to consider the incomplete information case where the reward structure and/or transition probabilities are imprecise. For instance, stochastic games with a single non-absorbing state where the payoff follows a given probability distribution were examined by Sorin (1984,1985). Najim et al.(2001) studied the optimization of the limiting average payoff of a zero-sum stochastic game with unknown transition probabilities and average payoffs. Luque-Vasquez and Minjarez-Sosa (2013) and Minjarez-Sosa and Vega-Amaya(2009) considered a zero-sum stochastic game where the payoffs are possibly unbounded. Cheng et al.(2016) developed two approximation methods to solve a two-person zero-sum stochastic game where the payoff matrix entries are independent and normally distributed. Kardes et al.(2011), Kardes(2014) and Rosenberg et al.(2004) analyzed equilibrium points for stochastic games with uncertain payoffs and/or transition probabilities. In all these studies, the state of the system is perfectly observable to each agent.

2.2 Partially Observable Stochastic Games

POSGs are stochastic games with imperfect information where the state of the system is partially observed. A POSG can be transformed to a normal-form game and theoretically solved by iterated elimination of dominated strategies; however, this representation is often too large and not computationally feasible (Hansen et al. 2004). Also, it is impossible to transform POSGs into completely observable stochastic games over belief states, analogous to how a POMDP is solved by transforming it into a MDP over belief states (Hansen et al. 2004). At each stage, each agent will receive a unique observation, leading to various different, possibly conflicting belief states (Emery-Montemerlo et al. 2004). Moreover, in a multi-agent system, each agent must also consider other agents’ beliefs to reason their actions. Even worse, each agent must reason about the beliefs that other agents hold about each other’s beliefs, leading to infinitely nested beliefs. As a result, POSGs suffer significant computational challenges: finding an optimal solution or even computing solutions with absolutely bounded error to common-payoff POSGs is $NEXP$-complete (Bernstein et al. 2002; Rabinovich et al. 2012); Goldsmith and Mundhenk(2008) further showed that the complexity of competitive POSGs can rise to $NEXP^{NP}$ (problems are solvable by a $NEXP$ machine using an $NP$ set as an oracle).

There are no known tractable algorithms for computing optimal (or even reasonable) policies for genera-sum POSGs. Nevertheless, the AI community has made tremendous progress for special classes of POSGs. For example, for two-agent zero-sum POSGs, Ghosh et al.(2004) and Saha (2014) showed that the POSG can be transformed to a stochastic game if both agents share a single observation process. Wiggers et al.(2016) analyzed the structural properties of value functions for zero-sum POSGs. However, algorithms for determining optimal policies for zero-sum POSGs are still rare. For common-payoff POSGs, also called decentralized POMDP, numerous exact and approximation algorithms have been developed (see reviews in Seuken and Zilberstein 2005; Oliehoek 2012; Oliehoek and Amato 2016).

The leader-follower POSG in Chang et al. (2015a, 2015b) is a general-sum POSG where the leader-follower relationship makes the POSG both theoretically and computationally attractive. Assuming each agent uses finite-memory policies and has complete knowledge of the game setup, there is a finite-dimensional sufficient statistic (the belief state) that consolidates all available information history for each agent to make a decision. The belief state is not defined on the state space; rather, it is an agent’s belief over the set of all possible finite information histories of the other agent. Consequently, an agent can infer via its belief state both the system’s state and the other agent’s action. The existence of the sufficient statistics further allows for transforming the general-sum POSG to special structured POMDPs. This model is applied to assess the value of misinformation and disinformation in modern warfare (Chang et al. 2019). While the existing work assumes that each agent has complete knowledge of the game setup (e.g., reward structure, perfect rationality), this paper examines the case where the leader has limited knowledge of the follower (i.e., POSG with incomplete information).

Another related model is called Interactive POMDPs (Gmytrasiewicz and Doshi 2004). The Interactive POMDP (I-POMDP) is another generalization of POMDPs to multi-agent systems. In this framework, agents are described by a class of possible models, and these agent models are included in the definition of “interactive states”. An agent’s belief over these interactive states is a sufficient statistic. However, an agent’s belief is also a component of the other agents’ models. Thus, these beliefs are infinitely nested, making the problem computationally complex. Existing approximation solution techniques include policy iteration (Sonu and Doshi 2012), point based value iteration (Doshi and Perez 2008), and interactive particle filtering (Doshi and Gmytraslewicz 2009).

2.3 Partially Observable Markov Decision Processes with Imprecise Parameters

POMDPs with imprecise parameters were analyzed in Itoh and Nakamura (2007) under the notion of “second-order beliefs” which are beliefs in the imprecisely specified transition probabilities and observation probabilities. The authors determined a set of optimal policies, each of which is optimal to at least one of such second-order beliefs. Saghafian (2018) examined the structural results of ambiguous POMDPs using “$\alpha$-maxmin” expected utility. Within the “maxmin” criterion, Osogami (2015) proved that the value function can still be convex using the Loomis’ Minimax Theorem, given the uncertainty set of the POMDP model parameters is convex. Under the assumption of S-rectangularity, Rasouli and Saghafian (2018) defined a robust POMDP, developed dynamic programming equations for it, and showed a zero-sum POSG can be transformed to a robust POMDP. However, to our best knowledge, there are no general algorithms for POMDPs with imprecise parameters yet (under the criterion of “maxmin”). We further remark that none of these assumptions necessarily holds in our problem (especially when facing with unknown adversaries in a security context) and we study general-sum leader-follower POSGs.

3 The Worst-Case Analysis Modeling

We consider a partially observable stochastic game with a leader ($L$) and a follower ($F$). At each stage, the leader first selects an action, and then the follower determines its action. Once the two actions are selected, the reward for each agent is realized and the system transitions to a new state according to a given probability. However, the leader’s information on the adversarial follower is very limited. Specifically,

1. (i)

Decision Horizon: The decision epochs are $t=0,1,2,...,T$ where $T\leq\infty$. 2. (ii)

State Space: the leader’s state $s_{t}^{L}\in S^{L}$ and the follower’s state $s_{t}^{F}\in S^{F}$. The state spaces $S^{L}$ and $S^{F}$ are assumed to be finite. Each agent knows its own state but may have only inaccurate observations of the other agent’s state. 3. (iii)

Action Space: at each stage, the leader selects $a^{L}_{t}\in A^{L}$ and the follower selects its (true) action $\tilde{a}_{t}^{F}\in A^{F}$, assuming the action spaces $A^{L}$ and $A^{F}$ are finite. How the follower selects its action $\tilde{a}_{t}^{F}$ is unknown to the leader, and the follower is possibly irrational. 4. (iv)

Observation Space: at each stage, the leader receives noisy observation $z_{t}^{L}\in Z^{L}$ of the follower’s state $s^{F}_{t}$, where the leader’s observation space $Z^{L}$ is finite. What observations $z^{F}_{t}$ that the follower may collect is unclear to the leader. 5. (v)

The State Transition and Observation Probabilities: The conditional probability for the leader $P(z^{L}_{t+1},s_{t+1}|s_{t},a_{t})$ is assumed given. The follower’s observation probabilities are unknown. 6. (vi)

Reward Structure: the leader’s scalar reward is $r^{L}(s_{t},a_{t})$, given the state pair $s_{t}=(s_{t}^{L},s_{t}^{F}),s_{t}^{L}\in S^{L},s_{t}^{F}\in S^{F}$ and action pair $a_{t}=(a_{t}^{L},a_{t}^{F}),a^{L}_{t}\in A^{L},a^{F}_{t}\in A^{F}$; the follower’s reward $r^{F}(s_{t},a_{t})$ is unknown.

The problem objective is to determine the best worst-case value function for the leader and its associated policy, given the above assumptions.

Because the follower only chooses $\tilde{a}^{F}_{t}$ after the selection of the leader’s action and $\tilde{a}^{F}_{t}$ may not be observable, the worst-case analysis assumes that at each stage $t$, the leader

1. (i)

first predicts the follower’s worst response action $a_{t}^{F}\in A^{F}$ for each $a^{L}_{t}$ (note, $a_{t}^{F}\neq\tilde{a}_{t}^{F}$); 2. (ii)

determines its best leader’s action $a^{L}_{t}\in A^{L}$ (see Figure 1).

Thus, at time $t$, the leader’s information history is $\zeta_{t}^{L}=\{s_{t}^{L},...,s_{0}^{L},z_{t}^{L},...,z_{1}^{L},a_{t-1},...,\\ a_{0},x_{0}^{L}\}$, where $x_{0}^{L}=\{P(s_{0}^{F}),s_{0}^{F}\in S^{F}\}$ is the leader’s prior probability mass vector over $S^{F}$ and $a_{t}=(a_{t}^{L},a_{t}^{F})$. Note that the “predicted” follower’s action $a_{t}^{F}$ is also a decision variable of the leader, same as $a_{t}^{L}$, and $\zeta^{L}_{t+1}=\{z^{L}_{t+1},s^{L}_{t+1},a_{t},\zeta^{L}_{t}\}$.

The criterion we consider $v_{0}^{L}(\zeta_{0}^{L})$ is the expected total discounted reward accrued over horizon $T$. Namely, $v_{0}^{L}(\zeta_{0}^{L})=E\{\sum_{t=0}^{T}\beta^{t}r^{L}(s_{t},a_{t})|\zeta_{0}^{L}\}$ for the finite horizon case and $v_{0}^{L}(\zeta_{0}^{L})=E\{\sum_{t=0}^{\infty}\beta^{t}r^{L}(s_{t},a_{t})|\zeta_{0}^{L}\}$ for the infinite horizon case, where $E\{.|\zeta_{0}^{L}\}$ is the expectation operator conditioned on $\zeta_{0}^{L}$, and $\beta\geq 0$ is the discount factor. We assume $\beta<1$ for the infinite horizon case in order to ensure that $E\{\sum_{t=0}^{\infty}\beta^{t}r^{L}(s_{t},a_{t})|\zeta_{0}^{L}\}$ is well defined. The problem objective is to determine a policy pair for the leader $(\pi^{L,*},\pi^{F,*}):\{\zeta_{t}^{L}\}\rightarrow A^{L}\times A^{F}$ such that

[TABLE]

where $\Pi^{k}$ is the policy space of agent $k,k\in\{L,F\}$.

The POSG under the worst-case scenario results in a single-agent dynamic decision making problem: the leader determines both $a^{L}_{t}$ and the worst-case action $a^{F}_{t}$ on the basis of $\zeta^{L}_{t}$. Thus, this model is fundamentally different from a zero-sum POSG (which itself is a challenging problem; see Section 2). In zero-sum POSGs, agent $k$ makes its decision based on its own private information history $\zeta^{k}_{t}$. That is, assuming the follower is rational in the zero-sum POSG, the follower will select its (true) action $\tilde{a}_{t}^{F}$ on the basis of $\zeta^{F}_{t}$, whereas in the worst-case analysis, the leader has no knowledge of $\zeta^{F}_{t}$ and “predicts” the worst-case action $a^{F}_{t}$ based on the leader’s knowledge $\zeta_{t}^{L},\zeta_{t}^{L}\neq\zeta_{t}^{F}$. Secondly, the follower may not be perfectly rational in our worst-case model.

Furthermore, knowing the exact information history that an adversary has is hard: $\zeta_{t}^{F}$ can be $\{s_{t}^{F},...,s_{0}^{F},z_{t}^{F},...,z_{1}^{F},\tilde{a}_{t-1}^{F},...,\tilde{a}_{0}^{F},x_{0}^{F}\},x_{0}^{F}=\{P(s_{0}^{L}),s_{0}^{L}\in S^{L}\}$ or other possibly unknown forms. It is also unclear how the follower will utilize $\zeta_{t}^{F}$ to make its decision (e.g., the follower is myopic or irrational). The worst-case analysis requires no knowledge of (i) the follower’s private information history $\zeta_{t}^{F}$, and (ii) how the follower selects its action $\tilde{a}_{t}^{F}$. Instead, the worst-case modeling analyzes from the leader’s perspective; namely, at epoch $t$, the leader predicts the follower’s worst response action and selects its best action all based on its own information history $\zeta_{t}^{L}$. As a result, Eq. (1) is the baseline performance of the leader; i.e., $\forall\pi^{F}\in\Pi^{F},v_{0}^{L,\pi^{L,*},\pi^{F,*}}(\zeta_{0}^{L})\leq v_{0}^{L,\pi^{L,*},\pi^{F}}(\zeta_{0}^{L})$ for each given $\zeta_{0}^{L}$, which is exactly the objective of the “worst-case analysis” of this paper.

4 Structural Results

Let $v_{t}^{L}(\zeta_{t}^{L})$ be the maximal value of the worst-case expected total discounted reward to be accrued from epoch $t$ until $T$, given information history $\zeta_{t}^{L}$, then $v_{t}^{L}(\zeta_{t}^{L})$ can be described recursively by

[TABLE]

Let $x_{t}^{L}=\{x_{t}^{L}(s^{F}_{t}),s^{F}_{t}\in S^{F}\}$, where $x_{t}^{L}(s^{F}_{t})=P(s^{F}_{t}|\zeta_{t}^{L})$. Thus, $x_{t}^{L}$ is a “belief” array indicating the leader’s inference about the follower’s state $s^{F}_{t}$. Furthermore, the leader’s belief process $x^{L}_{t}$ is a controlled Markov process as there is a function $\lambda^{L}$ depending on $z_{t+1}^{L},s^{L}_{t+1},s^{L}_{t},x^{L}_{t}$ and $a_{t}=(a^{L}_{t},a^{F}_{t})$ such that

[TABLE]

where

[TABLE]

Denote the bottom of Eq. (4) by

[TABLE]

assuming it is non-zero. Let $V$ be the set of all bounded, real-valued functions on $S^{L}\times X^{L}$ having supremum norm $||v||=\sup\{|v(s^{L},x^{L})|:s^{L}\in S^{L},x^{L}\in X^{L}\}$, where $X^{L}=\{\sum_{s^{F}}x^{L}(s^{F})=1,x^{L}(s^{F})\geq 0\}$. Then $(V,||.||)$ is a Banach space. We say a real-valued function $f(s^{L},x^{L})$ for a fixed $s^{L}$ is piecewise linear on $X^{L}$ if there exists a set $\Gamma(s^{L})$ depending on $s^{L}$, $|\Gamma(s^{L})|<\infty$ such that: $\forall x^{L}\in X^{L}$, there is a $\gamma\in\Gamma(s^{L})$ satisfying $f(s^{L},x^{L})=x^{L}\gamma$, where $x^{L}\gamma=\sum_{s^{F}}x^{L}(s^{F})\gamma(s^{F})$. Now, define the operator $H:V\rightarrow V$ as

[TABLE]

where $x^{L}r^{L}(s^{L},a)=\sum_{s^{F}}x^{L}(s^{F})r^{L}(s^{L},s^{F},a)$.

Proposition 1**.**

[TABLE]

Thus, $v_{t}^{L}$ is dependent on $\zeta_{t}^{L}$ only through $(s^{L}_{t},x_{t}^{L})$, and $(s^{L}_{t},x_{t}^{L})$ is a sufficient statistic for $v_{t}^{L}$. Furthermore, if $v$ is piecewise linear in $x^{L}$, then $Hv$ is also piecewise linear in $x^{L}$.

Proof.

The first part follows by mathematical induction and the fact that both $\sigma^{L}(z^{L}_{t+1},s^{L}_{t+1},s^{L}_{t},x_{t}^{L},a_{t})$ and $\lambda^{L}(z^{L}_{t+1},s^{L}_{t+1},s^{L}_{t},x_{t}^{L},a_{t})$ are functions of $(s^{L}_{t},x_{t}^{L})$. For the second part, assume $v$ is piecewise linear. Equivalently, there exists a finite set $\Gamma(s^{L^{\prime}})$ such that

[TABLE]

where the function $l(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)$ defines the index of the $\gamma$ vector corresponding to $\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)$. We say $\gamma^{{}^{\prime}a}\in\Gamma^{{}^{\prime}}(s^{L},a)$, if $\gamma^{{}^{\prime}a}$ is of the form

[TABLE]

where $\gamma\in\Gamma(s^{L^{\prime}})$. Note,

[TABLE]

where $\Gamma^{\prime}(s^{L})=\cup_{a\in A}\Gamma^{\prime}(s^{L},a)$. Hence, $Hv$ is piecewise linear in $x^{L}$ for $|A^{L}\times A^{F}|<\infty$. ∎∎

We remark that this sufficient statistic is more computationally tractable than other sufficient statistics presented in the POSG literature. In a multi-agent system, not only do agents have to infer the underlying system state, but also must consider other agents’ beliefs in order to infer the other agents’ actions. As a result, the existing sufficient statistics are often complex and in high dimensions: they usually include a belief over the other agent’s (finite) information history $\{P(\zeta^{F,\tau}_{t}|\zeta^{L}_{t})\}$ where $\zeta^{F,\tau}_{t}=\{s_{t}^{F},...,s_{t-\tau+1}^{F},z_{t}^{F},...,z_{t-\tau+1}^{F},\tilde{a}_{t-1}^{F},...,\tilde{a}_{t-\tau}^{F}\}$ under the finite-memory assumption (Chang et al. 2015a), or a belief over the other agent’s complete model (Gmytrasiewicz and Doshi 2004). The worst-case modeling requires no such assumptions and the leader “predicts” the follower’s worst-case action based on its own knowledge, resulting in a much simpler sufficient statistic and a more computationally attractive problem.

The worst-case analysis transforms a general-sum POSG to a single agent problem, and hence it is relatively simple compared to multi-agent problems. Further, Eq. (6) is very similar to a POMDP with the newly defined $\lambda^{L}$ and $\sigma^{L}$, suggesting that solution procedures for the POMDP may be relevant to our problem. Unfortunately, the fact that the value function of a POMDP is both piecewise linearity and convexity forms the basis for existing POMDP algorithms (Sondik 1971; Cheng 1988; Pineau et al. 2003; Shani et al. 2013). While Proposition 1 guarantees that the operator $H$ in Eq. (6) preserves piecewise linearity, convexity will not be preserved, as illustrated by the following example. The non-convexity issue significantly limits the usefulness of solution procedures for the POMDP, and also makes our problem unique from the POMDP.

Consider $|S^{F}|=|A^{L}|=2$ and let $v_{T}^{L}=0$. Fix a $s^{L}\in S^{L}$, then $v_{T-1}^{L}(s^{L},x^{L})=\max_{a^{L}}\min_{a^{F}}\left\{\sum_{s^{F}}x^{L}(s^{F})r^{L}(s^{L},s^{F},a^{L},a^{F})\right\}=\max\{f_{1}(x^{L}),f_{2}(x^{L})\}$ where $f_{1}(x^{L})=\min\{4.6x_{1}^{L}+7.6x_{2}^{L},8.2x_{1}^{L}+x_{2}^{L}\}$ (black line), $f_{2}(x^{L})=\min\{1.8x_{1}^{L}+3.6x_{2}^{L},0.6x_{1}^{L}+5.2x_{2}^{L}\}$ (blue line), and $(x_{1}^{L},x_{2}^{L})$ is the leader’s belief vector over the follower’s state, $x_{1}^{L}+x_{2}^{L}=1,x_{1}^{L},x_{2}^{L}\geq 0$. Clearly, $v_{T-1}^{L}$ is non-convex in $x^{L}$ (red line). See Fig. 2.

For the infinite planning horizon, it is easy to show that

Proposition 2**.**

$\forall 0\leq\beta<1$, the operator $H$ is a contraction mapping on $V$ having modulus $\beta$.

Proof.

Let $u,v\in V$, fix $s^{L},x^{L}$ and assume $Hu(s^{L},x^{L})\geq Hv(s^{L},x^{L})$, then

[TABLE]

For any $a^{L}$, let a^{F,*}\in\arg\min\bigg{[}x^{L}r^{L}(s^{L},a)+\beta\sum_{z^{L^{\prime}}}\sum_{s^{L^{\prime}}}\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)\\ v(s^{L^{\prime}},\lambda^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a))\bigg{]} then

[TABLE]

Thus, $0\leq[Hu](s^{L},x^{L})-[Hv](s^{L},x^{L})\leq\beta||u-v||$. Repeating this argument in the case that $0\leq[Hv](s^{L},x^{L})-[Hu](s^{L},x^{L})$ shows that $|[Hu](s^{L},x^{L})-[Hv](s^{L},x^{L})|\leq\beta||u-v||$. Taking the supremum over $s^{L}$ and $x^{L}$ gives $||Hu-Hv||\leq\beta||u-v||.$ ∎∎

As a result, there is a unique fixed point $v^{*}\in V$ such that $Hv^{*}=v^{*}$. Let the sequence $\{v_{n}\}$ be defined as $v_{n+1}=Hv_{n}$, then $\lim_{n\rightarrow\infty}||v^{*}-v_{n}||=0$, given $\forall v_{0}\in V$. Moreover, $v^{*}$ is continuous. However, it is not guaranteed that $v^{*}$ is convex or concave.

We remark that the worst-case model can also be viewed as a POMDP with imprecise parameters, where the leader’s reward $r^{L}$ and dynamics $P(z_{t+1}^{L},s_{t+1}|s_{t},a_{t})$ are imprecise due to the unknown $a_{t}^{F}$. However, to our best knowledge, general algorithms do not exist for such problems with the “maxmin” criterion (the existing literature focuses on imposing conditions to avoid the non-convexity issue; see Section 2). In this paper, we propose a novel computationally attractive solution procedure without these conditions that could generate a lower bound for the finite-horizon value function with quantified error bound, and determine its associated control policy. Research on the infinite horizon case is a topic for future research.

5 Lower Bound Solution Approach and Policy Determination

We motivate the lower bound solution approach by a rewrite of Eq. (6) to two steps: the “min” step and the “max” step. Specifically, $\forall a^{L}\in A^{L}$, let the operator $H^{a^{L}}:V\rightarrow V$ be

[TABLE]

then,

[TABLE]

For the “min” step, assume $\tilde{v}^{L}$ is a piecewise linear and concave approximation of $v^{L}$ satisfying $\tilde{v}^{L}\leq v^{L}$ (i.e., $\tilde{v}^{L}(s^{L},x^{L})\leq v^{L}(s^{L},x^{L}),\forall s^{L}\in S^{L},x^{L}\in X^{L}$). Namely, there is a finite set $\tilde{\Gamma}(s^{L})$ such that $\tilde{v}^{L}(s^{L},x^{L})=\min\{x^{L}\gamma:\gamma\in\tilde{\Gamma}(s^{L})\}$. Then, pick any $a^{L}\in A^{L}$,

[TABLE]

where $\gamma\in G(s^{L},a^{L})$ if $\gamma=r^{L}(s,a)+\beta\sum_{z^{L^{\prime}}}\sum_{s^{{}^{\prime}}}P(z^{L^{\prime}},s^{\prime}|s,a)\gamma^{\prime}(s^{F^{\prime}})$. Thus, $H^{a^{L}}\tilde{v}^{L}$ is piecewise linear and concave, and the “min” step is a POMDP step.

For the “max” step, since $Hv=\max_{a^{L}\in A^{L}}[H^{a^{L}}v]$, we have the following:

Theorem 1**.**

There is a finite set of arrays $\Gamma(s^{L})=\{\gamma^{k_{1},k_{2}}\}_{k_{1},k_{2}\geq 0}$ only depends on $s^{L}$ such that

[TABLE]

Proof.

See the paragraphs above the Theorem. ∎∎

Policy Determination: Each element of $\Gamma(s^{L})$ is associated with a pair of action $(a^{L},a^{F})$. Denote $\Gamma(s^{L},k_{1})=\{\gamma^{k_{1}^{\prime},k_{2}^{\prime}}:\gamma^{k_{1}^{\prime},k_{2}^{\prime}}\in\Gamma(s^{L}),k_{1}^{\prime}=k_{1}\}$. Each set $\Gamma(s^{L},k_{1})$ corresponds to a leader’s action $a^{L}$, and each vector $\gamma\in\Gamma(s^{L},k_{1})$ is associated with a follower’s worst-case action $a^{F}$. The best worst-case policy for $H\tilde{v}$ can thus be determined by the following steps:

1. (i)

determine $(s^{L},x^{L})$; 2. (ii)

for each $k_{1}$, find $\gamma^{k_{1},*}$ in $\arg\min\{x^{L}\gamma:\gamma\in\Gamma(s^{L},k_{1})\}$, and let $\Theta(s^{L})=\cup_{k_{1}}\{\gamma^{k_{1},*}\}$; 3. (iii)

determine $\gamma^{*,*}$ in $\arg\max\{x^{L}\gamma:\gamma\in\Theta(s^{L})\}$; 4. (iv)

select the action pair associated with $\gamma^{*,*}$.

In a nutshell, at time t, if there is a $\tilde{v}^{L}_{t+1}$ being a piecewise linear concave approximation of $v_{t+1}^{L}$, $\tilde{v}^{L}_{t+1}\leq v_{t+1}^{L}$, we can determine $H^{a^{L}}\tilde{v}^{L}_{t+1}$ and $H\tilde{v}^{L}_{t+1}$ by POMDP algorithms and Theorem 1, respectively. Moreover,

Proposition 3**.**

If $\tilde{v}^{L}_{t+1}\leq v_{t+1}^{L}$, then $\bar{v}_{t}^{L}=H\tilde{v}^{L}_{t+1}\leq v^{L}_{t}=Hv_{t+1}^{L}$.

Proof.

This is due to the definition of $H$ and $\sigma^{L}(z^{L^{\prime}},s^{L^{\prime}},s^{L},x^{L},a)\geq 0$. ∎∎

As $\bar{v}_{t}^{L}$ is again non-convex, we then need to find another $\tilde{v}^{L}_{t}$ to best approximate $\bar{v}_{t}^{L}$ and repeat the process for time $t-1$. Fig. 3 presents a three-step procedure to implement this idea for the finite planning horizon problem.

Specifically, given $\tilde{v}_{t+1}^{L}$ being a piecewise linear concave approximation of $v^{L}_{t+1}$, $\bar{v}^{a^{L}}_{t}(s^{L},x^{L})=[H^{a^{L}}\tilde{v}_{t+1}](s^{L},x^{L})=\min\{x^{L}\gamma:\gamma\in G(s^{L},a^{L})\}$ is a POMDP step for $\forall a^{L}\in A^{L}$. As in all POMDP problems, the set $G(s^{L},a^{L})$ may contain many redundant $\gamma$-vectors never used in determining $\bar{v}^{a^{L}}_{t}$. The PURGE-step is to remove all redundant $\gamma$-vectors in each $G(s^{L},a^{L})$, which can be accomplished by the PURGE operator in the POMDP literature. According to Theorem 1 and $H\tilde{v}_{t+1}=\max_{a^{L}\in A^{L}}[H^{a^{L}}\tilde{v}_{t+1}]$, we could set $\Gamma_{t}(s^{L})=G(s^{L})$ where $G(s^{L})=\cup_{a^{L}\in A^{L}}G(s^{L},a^{L})$; however, the resulting set can be very large. The DOMINANCE-step is to determine the set $\Gamma_{t}(s^{L})$ that has the smallest cardinality and satisfies $\max_{k_{1}}\min_{k_{2}}\{x^{L}\gamma^{k_{1},k_{2}}:\gamma\in\Gamma_{t}(s^{L})\}=\max_{a^{L}}\min_{k_{2}}\{x^{L}\gamma^{a^{L},k_{2}}:\gamma^{a^{L},k_{2}}\in G(s^{L})\}$ for computational advantage. The DOMINANCE operator is designed via computational geometry and mixed integer programs (MIPs). Because the resulting $\bar{v}_{t}^{L}=H\tilde{v}_{t+1}^{L}$ is again non-convex, the APPROXIMATION-step approximates $\bar{v}_{t}^{L}$ with quantified error by a piecewise linear and concave function $\tilde{v}_{t}^{L}$ satisfying $\tilde{v}_{t}^{L}\leq\bar{v}_{t}^{L}$ for the next iteration.

Performing all required operations and approximation, we have developed a backward recursive algorithm for determining a lower bound of the leader’s best worst-case value function and its associated control policy for the finite-horizon POSG. The pseudocode of the entire procedure is summarized in Algorithm 1. The rest of the paper presents each step in more detail.

6 PURGE Operation

Assume a piecewise linear and concave function $\tilde{v}_{t+1}^{L}$ satisfying $\tilde{v}_{t+1}^{L}\leq v_{t+1}^{L}$ is given. That is, there is a set $\tilde{\Gamma}_{t+1}(s^{L})$ for each $s^{L}\in S^{L}$ where $\tilde{v}_{t+1}^{L}(s^{L},x^{L})=\min\{x^{L}\gamma:\gamma\in\tilde{\Gamma}_{t+1}(s^{L})\}$. For any $a^{L}\in A^{L}$, we have shown that $\bar{v}^{a^{L}}_{t}(s^{L},x^{L})=[H^{a^{L}}\tilde{v}_{t+1}^{L}](s^{L},x^{L})=\min\{x^{L}\gamma:\gamma\in G(s^{L},a^{L})\}$ where

[TABLE]

A large number of $\gamma$-vectors could be generated in this step, however, only a small number of these vectors define $\bar{v}^{a^{L}}_{t}$. A $\gamma\in G(s^{L},a^{L})$ is redundant if and only if for all $x^{L}\in X^{L}$, there is a $\gamma^{\prime}\in G(s^{L},a^{L}),\gamma^{\prime}\neq\gamma$ such that $x^{L}\gamma^{\prime}\leq x^{L}\gamma$; a $\gamma\in G(s^{L},a^{L})$ is referred to as a defining vector for $\bar{v}^{a^{L}}_{t}$ if there is a belief point $x^{*}\in X^{L}$ such that $\bar{v}^{a^{L}}_{t}(s^{L},x^{*})=x^{*}\gamma$ and these belief points are called witness points (Lin et al. 2004). The PURGE operators in Cassandra et al.(1997), Lin et al.(2004) (and many others) in the POMDP literature can be employed to efficiently remove redundant vectors from $G(s^{L},a^{L})$. We adopt the PURGE operator in Lin et al.(2004) in our illustrative example.

7 DOMINANCE Operation

We now determine $\Gamma_{t}(s^{L})\subseteq G(s^{L})$ where $G(s^{L})=\cup_{a^{L}\in A^{L}}G(s^{L},a^{L}),\forall s^{L}\in S^{L}$ by extending the notion of redundancy of a $\gamma$-vector to a set. For a given $s^{L}\in S^{L}$, a set $G(s^{L},a^{L})$ is dominated by $\Gamma_{t}(s^{L})$ on $X^{L}$ if and only if $\forall x^{L}\in X^{L}$, there is always a set $G(s^{L},a^{L^{\prime}})$ in $\Gamma_{t}(s^{L})$ such that $\bar{v}^{a^{L}}_{t}(s^{L},x^{L})\leq\bar{v}^{a^{L^{\prime}}}_{t}(s^{L},x^{L})$ where $\bar{v}_{t}^{a^{L}}=\min\{x^{L}\gamma:\gamma\in G(s^{L},a^{L})\}$; a set $G(s^{L},a^{L})$ is referred to as supporting if there is at least one belief point $x^{\prime}\in X^{L}$ such that $\bar{v}_{t}^{L}(s^{L},x^{\prime})=\max_{k_{1}}\min_{k_{2}}\{x^{\prime}\gamma^{k_{1},k_{2}}:\gamma^{k_{1},k_{2}}\in\Gamma(s^{L})\}=\min\{x^{\prime}\gamma:\gamma\in G(s^{L},a^{L})\}$. For example, both sets $G(s^{L},a^{L}_{1})$ and $G(s^{L},a^{L}_{4})$ in Fig. 4 are dominated sets, while sets $G(s^{L},a^{L}_{2})$ and $G(s^{L},a^{L}_{3})$ are supporting for $\bar{v}_{t}^{L}$. We seek to remove all dominated sets in order to define $\Gamma_{t}$ efficiently.

Let the DOMINANCE operator be that $\Gamma(s^{L})=\mbox{DOMINANCE}(G(s^{L}))$ only contains supporting sets of $\bar{v}_{t}^{L}$. We present a two-step procedure for the DOMINANCE operator. We say a set $G(s^{L},a^{L})$ is pair-wise dominated by a set $G(s^{L},a^{L^{\prime}})$ if and only if $\forall x^{L}\in X^{L}$, $\bar{v}^{a^{L}}(s^{L},x^{L})\leq\bar{v}^{a^{L^{\prime}}}(s^{L},x^{L})$. For example, set $G(s^{L},a^{L}_{4})$ is pair-wise dominated by the set $G(s^{L},a^{L}_{1})$ in Fig. 4.

The first step is to build a superset $\Gamma_{t}^{c}(s^{L})\supseteq\Gamma_{t}(s^{L})$, where for any $G(s^{L},a^{L})$ in $\Gamma_{t}^{c}(s^{L})$, there is no $G(s^{L},a^{L^{\prime}})$ in $\Gamma_{t}^{c}(s^{L})$ pair-wise dominating $G(s^{L},a^{L})$. We show the pairwise dominance relationship between two sets in $G(s^{L})$ can be determined efficiently by a sequence of linear programs (LPs), employing a dual relationship between hyperplanes and points. However, a set $G(s^{L},a^{L})$ in $\Gamma_{t}^{c}(s^{L})$ could still be a dominated set, such as $G(s^{L},a^{L}_{1})$ in Fig. 4. The second step employs a MIP to further remove all dominated sets in $\Gamma_{t}^{c}(s^{L})$. The first step is to substantially decrease the number and the size of MIPs encountered in the second step.

7.1 Determine the Superset $\Gamma_{t}^{c}(s^{L})\supseteq\Gamma_{t}(s^{L})$

In computational geometry, the dual of a hyperplane $p(u)=\sum_{i=1}^{n-1}p_{i}u_{i}+p_{n}$ in the primal $R^{n}$ space is the point $p^{*}=(p_{1},...p_{n})\in R^{n}$, and the dual of a point $p=(p_{1},...p_{n})\in R^{n}$ is a hyperplane $p^{*}(u)=\sum_{i=1}^{n-1}p_{i}u_{i}+p_{n}$. The lower envelope of a given set of hyperplanes $\{p^{k}(u)=\sum_{i=1}^{n-1}p^{k}_{i}u_{i}+p^{k}_{n}\}_{k\in K}$ is the piecewise linear and concave function $\underline{p}(u)=\min_{k\in K}\{\sum_{i=1}^{n-1}p^{k}_{i}u_{i}+p^{k}_{n}\}$, whereas the upper envelope of the given set of hyperplanes is the piecewise linear and convex function $\bar{p}(u)=\max_{k\in K}\{\sum_{i=1}^{n-1}p^{k}_{i}u_{i}+p^{k}_{n}\}$. Each of the hyperplanes on the upper envelope in the primal space corresponds to a vertice of the upper convex hull (with respect to the $p_{n}$-axis) in the dual space (de Berg et al. 2008; Zhang 2010). We need the following definitions in Zhang (2010) to proceed.

Given a set $\Omega\in R^{|S^{F}|}$ of points, the convex hull is the set $Co(\Omega)\equiv\{\sum_{j=1}\lambda_{j}w_{j}:\sum_{j=1}\lambda_{j}=1\mbox{ and }w_{j}\in\Omega,\lambda_{j}\geq 0,\forall j\}$. The surface of the convex hull with negative outernormal directions, the negative convex hull ($NCo$), is the set $NCo(\Omega)\equiv cl(\{w\in Co(\Omega):\exists x\in X^{+},xw\leq x\gamma,\forall\gamma\in Co(\Omega)\})$, where $X^{+}=\{x\in X\mbox{ and }x_{i}>0,\forall i\}$, and $cl(B)$ is the closure of $B$. Then,

Lemma 1**.**

Suppose that $\Omega=G(s^{L},a^{L})\in R^{|S^{F}|}$ is closed and bounded, hence, compacted. For any given $a^{L}$ and $s^{L}$, the piecewise linear and concave function $\bar{v}^{a^{L}}(s^{L},x^{L})=\min\{x^{L}\gamma:\gamma\in G(s^{L},a^{L})\},x^{L}\in X^{L}$, is dual to the set $NCo(\Omega)$. Namely, for any $\hat{x}^{L}\in X^{L}$, there exists a $\hat{\gamma}\in NCo(\Omega)$ such that $\bar{v}^{a^{L}}(s^{L},\hat{x}^{L})=\hat{x}^{L}\hat{\gamma}$, and conversely, for any $\hat{\gamma}\in NCo(\Omega)$, there is a $\hat{x}^{L}\in X^{L}$ such that $\bar{v}^{a^{L}}(s^{L},\hat{x}^{L})=\hat{x}^{L}\hat{\gamma}$.

Proof.

It follows the proof of Lemma 1 in Zhang (2010). ∎∎

We now determine whether a set $G(s^{L},a^{L})$ is pair-wise dominated by a set $G(s^{L},a^{L^{\prime}})$ based on the geometric relationship. Without loss of generality, assume $s^{L}$ is given and there is $x^{0}$ such that $\bar{v}^{a^{L}}_{t}(s^{L},x^{0})\leq\bar{v}^{a^{L^{\prime}}}_{t}(s^{L},x^{0})$. Pick any $\gamma\in G(s^{L},a^{L^{\prime}})$ and define the set $\Phi(\gamma)\equiv\{(\lambda_{1},...,\lambda_{n}):\gamma\geq\sum_{i=1}^{n}\lambda_{i}w^{i},w^{i}\in G(s^{L},a^{L}),\sum_{i=1}^{n}\lambda_{i}=1,\lambda_{i}\geq 0\}$. Proposition 4 shows that determining the dominance relationship is equivalent to check whether $\Phi(\gamma)$ is empty for every $\gamma\in G(s^{L},a^{L^{\prime}})$. The detailed pseudocode is summarized in Algorithm 2.

Proposition 4**.**

$G(s^{L},a^{L})$* is dominated by $G(s^{L},a^{L^{\prime}})$ if and only if for each $\gamma\in G(s^{L},a^{L^{\prime}})$, the set $\Phi(\gamma)$ is non-empty.*

Proof.

Assume for each $\gamma\in G(s^{L},a^{L^{\prime}})$, $\Phi(\gamma)$ is non-empty. Equivalently, $\forall\gamma\in G(s^{L},a^{L^{\prime}})$, there is $\omega_{\gamma}\in Co(G(s^{L},a^{L}))$ such that $\gamma\geq\omega_{\gamma}$. Pick $\forall\hat{x}^{L}\in X^{L}$ and let $\hat{\gamma}\in\arg\min\{\hat{x}^{L}\gamma:\gamma\in G(s^{L},a^{L^{\prime}})\}$. Thus, $\bar{v}^{a^{L^{\prime}}}_{t}(s^{L},\hat{x}^{L})=\hat{x}^{L}\hat{\gamma}\geq\hat{x}^{L}\omega_{\hat{\gamma}}$. Lemma 1 guarantees that there is a $\hat{\omega}\in NCo(G(s^{L},a^{L}))$ satisfying $\bar{v}^{a^{L}}_{t}(s^{L},\hat{x}^{L})=\hat{x}^{L}\hat{\omega}\leq\hat{x}^{L}\omega_{\hat{\gamma}}$. Thus, $\bar{v}^{a^{L}}_{t}(s^{L},x^{L})\leq\bar{v}^{a^{L^{\prime}}}_{t}(s^{L},x^{L}),\forall x^{L}\in X^{L}$ and the set $G(s^{L},a^{L})$ is dominated by the set $G(s^{L},a^{L^{\prime}})$.

Conversely, let $\gamma^{*}\in G(s^{L},a^{L^{\prime}})$ be the $\gamma$-vector such that $\Phi(\gamma^{*})=\emptyset$. Equivalently, $\gamma^{*}<\omega,\forall\omega\in NCo(G(s^{L},a^{L}))$. As every $\gamma$-vector in $G(s^{L},a^{L^{\prime}})$ is a defining vector for $\bar{v}^{a^{L^{\prime}}}$, there is $x^{*}\in X^{L}$ such that $\bar{v}^{a^{L^{\prime}}}(s^{L},x^{*})=x^{*}\gamma^{*}$. Lemma 1 further guarantees that there is a $\omega^{*}\in NCo(G(s^{L},a^{L}))$ satisfying $\bar{v}^{a^{L}}(s^{L},x^{*})=x^{*}\omega^{*}$. Thus, $\bar{v}^{a^{L^{\prime}}}(s^{L},x^{*})<\bar{v}^{a^{L}}(s^{L},x^{*})$. Since $\bar{v}^{a^{L}}_{t}(s^{L},x^{0})\leq\bar{v}^{a^{L^{\prime}}}_{t}(s^{L},x^{0})$ by assumption, both functions are continuous, and $X^{L}$ is connected, the two functions intersect over $X^{L}$. ∎∎

We remark that we also can determine the dominance on primal space $X^{L}$ by checking if $NCo(G(s^{L},a^{L}))\cap NCo(G(s^{L},a^{L^{\prime}}))\neq\emptyset$ in the dual space, given $s^{L}\in S^{L}$. See Proposition 5. Conversely, however, if $\bar{v}^{a^{L}}_{t}(s^{L},x^{L})$ and $\bar{v}^{a^{L^{\prime}}}_{t}(s^{L},x^{L})$ intersect on $X^{L}$, $NCo(G(s^{L},a^{L}))\cap NCo(G(s^{L},a^{L^{\prime}}))$ could be empty in the dual space. A counterexample is given Fig. 5.

Proposition 5**.**

$\forall s^{L}\in S^{L}$, if $NCo(G(s^{L},a^{L}))\cap NCo(G(s^{L},a^{L^{\prime}}))\neq\emptyset$, then $\bar{v}^{a^{L}}_{t}(s^{L},x^{L})$ and $\bar{v}^{a^{L^{\prime}}}_{t}(s^{L},x^{L})$ intersect on $X^{L}$.

Proof.

Pick $\forall\gamma^{*}\in NCo(G(s^{L},a^{L}))\cap NCo(G(s^{L},a^{L^{\prime}}))$. Let $x^{*}\in X^{L}$ such that $\bar{v}^{a^{L}}_{t}(s^{L},x^{*})=x^{*}\gamma^{*}$ per Lemma 1. The definition of $NCo$ and Lemma 1 guarantee $\bar{v}^{a^{L^{\prime}}}_{t}(s^{L},x^{*})\leq x^{*}\gamma^{*}$. The result follows by the assumption that $\bar{v}^{a^{L^{\prime}}}_{t}(s^{L},x^{0})\geq\bar{v}^{a^{L}}_{t}(s^{L},x^{0}).$ ∎∎

Determining the superset $\Gamma^{c}(s^{L})$ on the basis of pairwise dominance needs to consider each pair of action $a^{L},a^{L^{\prime}}\in A^{L}$ (the pseudocode is summarized in Algorithm 3). For each $s^{L}\in S^{L}$, the program initializes $\Gamma^{c}(s^{L})$ with the set $G(s^{L},a^{L,*})$, where $a^{L,*}\in\arg\max_{a^{L}\in A^{L}}\bar{v}^{a^{L}}_{t}(s^{L},x^{0})$ and $x^{0}\in X^{L}$ is randomly generated. Let $\Gamma^{c}(s^{L},k_{1})=\{\gamma^{k_{1}^{\prime},k_{2}^{\prime}}:\gamma^{k_{1}^{\prime},k_{2}^{\prime}}\in\Gamma^{c}(s^{L}),k_{1}^{\prime}=k_{1}\}$, $K_{1}(s^{L})$ be the number of $\Gamma^{c}(s^{L},k_{1})$ sets in $\Gamma^{c}(s^{L})$, and $K_{2}(s^{L},k_{1})$ be the number of $\gamma$-vectors in $\Gamma^{c}(s^{L},k_{1})$. The algorithm updates $\Gamma^{c}(s^{L}),K_{1}(s^{L})$, and $K_{2}(s^{L},k_{1})$ by the following: For each candidate set $G(s^{L},a^{L})$, the algorithm compares it with the existing sets in $\Gamma^{c}(s^{L})$. If $G(s^{L},a^{L})$ is pair-wise dominated by an existing set $\Gamma^{c}(s^{L},k_{1})$, then $G(s^{L},a^{L})$ will not be considered; Otherwise, $G(s^{L},a^{L})$ will be included in $\Gamma^{c}(s^{L})$ and any existing sets $\Gamma^{c}(s^{L},k_{1})$ that are dominated by $G(s^{L},a^{L})$ will be eliminated from $\Gamma^{c}(s^{L})$. Meanwhile, $K_{1}$ and $K_{2}$ are updated accordingly.

7.2 Determine the Set $\Gamma_{t}(s^{L})$

We now determine the set $\Gamma_{t}(s^{L})$ by further removing the dominated sets from $\Gamma_{t}^{c}(s^{L})$. Assume $s^{L}$ is given. $\forall x^{L}\in X^{L}$, let $z_{1}(x^{L})$ be the function value attained by the superset $\Gamma_{t}^{c}(s^{L})$, $z_{1}(x^{L})=\max_{k_{1}}\min_{k_{2}}\{x^{L}\gamma^{k_{1},k_{2}}:\gamma^{k_{1},k_{2}}\in\Gamma_{t}^{c}(s^{L})\}$, and $z_{2}(x^{L})$ be the value of function $\bar{v}^{k_{1}}$ attained by the set $\Gamma^{c}(s^{L},k_{1})$, $z_{2}(x^{L})=\min\{x^{L}\gamma:\gamma\in\Gamma^{c}(s^{L},k_{1})\}$. Let DOMINANCE_MIP($\Gamma^{c}(s^{L},k_{1}),\Gamma_{t}^{c}(s^{L})$) determine whether the set $\Gamma^{c}(s^{L},k_{1})$ is a dominated set, which can be evaluated via the following mixed integer program (1):

[TABLE]

where $M$ is a large positive number.

The objective function is to find the minimal gap between the two functions, $z_{1}(x^{L})$ and $z_{2}(x^{L})$. As $z_{2}(x^{L})$ is a piecewise linear and concave function on $X^{L}$, it can be easily determined by the first constraint. The second and the third constraints define $z_{1}$. For the purpose of explanation, $\forall k_{1}\in K_{1}(s^{L})$, we further introduce a variable $\eta^{k_{1}}$ as the minimum value attained by set $\Gamma_{t}^{c}(s^{L},k_{1})$, i.e., $\eta^{k_{1}}(x^{L})=\min\{x^{L}\gamma:\gamma\in\Gamma_{t}^{c}(s^{L},k_{1})\}$. Then, (i) $\eta^{k_{1}}\geq x^{L}\gamma^{k_{1},k_{2}}-M(1-\rho^{k_{1},k_{2}})$ and the multiple-choice constraint on $\rho^{k_{1},k_{2}}$s ensure that there is exactly one $\gamma\in\Gamma_{t}^{c}(s^{L},k_{1})$ selected to define $\eta^{k_{1}}$; (ii) $z_{1}\geq\eta^{k_{1}},\forall k_{1}$, by the definition of $z_{1}$. Hence, the combination of (i) and (ii) leads to the second constraint and variables $\eta^{k_{1}}$s can be omitted. The last equation ensures that the belief states are in a nonnegative simplex.

Clearly, if the objective value $u>0$, then $\Gamma_{t}^{c}(s^{L},k_{1})$ is not a supporting set for $\bar{v}_{t}^{L}$ and should be eliminated. We need to solve $K_{1}$ number of MIPs to finalize $\Gamma_{t}(s^{L})$. The pseudocode for determining $\Gamma_{t}(s^{L})$ from its superset $\Gamma_{t}^{c}(s^{L})$ is presented in Algorithm 4.

8 Piecewise Linear Concave Approximation

Given a set of $\gamma$-vectors $\Gamma_{t}$, the value function $\bar{v}_{t}^{L}(s^{L},x^{L})=\max_{k_{1}}\min_{k_{2}}\{x^{L}\gamma^{k_{1},k_{2}}:\gamma^{k_{1},k_{2}}\in\Gamma_{t}(s^{L})\}$ is piecewise linear but not concave. The iterative algorithm we developed requires a piecewise linear and concave function for the next iteration. We thus approximate $\bar{v}_{t}^{L}$ by a function $\tilde{v}_{t}^{L}$ satisfying the following conditions:

1. (i)

$\forall s^{L}\in S^{L},\tilde{v}_{t}^{L}(s^{L},x^{L})$ is a piecewise linear and concave on $X^{L}$; 2. (ii)

$\tilde{v}_{t}^{L}(s^{L},x^{L})\leq\bar{v}_{t}^{L}(s^{L},x^{L}),\forall s^{L}\in S^{L},x^{L}\in X^{L}$; 3. (iii)

the distance between $\tilde{v}_{t}^{L}$ and $\bar{v}_{t}^{L}$ is as small as possible, where we define the distance between two bounded functions $v^{1},v^{2}\in V$ as

[TABLE]

Equivalently, for each $s^{L}$, we want to determine a set $\tilde{\Gamma}_{t}(s^{L})$ so that $\tilde{v}_{t}^{L}(s^{L},x^{L})=\min\{x^{L}\gamma:\gamma\in\tilde{\Gamma}(s^{L})\}$ satisfies conditions (ii) and (iii). For computational efficiency, we consider the case where $\tilde{\Gamma}_{t}(s^{L})\subset\Gamma_{t}(s^{L})$ in this paper. We do acknowledge that $\tilde{v}_{t}^{L}(s^{L},x^{L})$ may be further improved by constructing $\gamma\notin\Gamma_{t}(s^{L})$ for some instances. Determining a general procedure for finding the best piecewise linear and concave approximation of an arbitrary piecewise linear function is an interesting research topic for the future. Furthermore, an advantage of selecting $\tilde{\Gamma}_{t}(s^{L})\subset\Gamma_{t}(s^{L})$ is that each $\gamma\in\tilde{\Gamma}_{t}(s^{L})$ is still associated with an action pair $a=(a^{L},a^{F})$. Thus, it is easy to explain and implement the policy associated with the lower bound $\tilde{v}_{t}^{L}(s^{L},x^{L})$.

We remark that the maximal gap between two functions $v^{1},v^{2}\in V$ for a given $s^{L}\in S^{L}$ must occur at (i) where two segments of $v^{1}$ (or $v^{2}$) intersect, or (ii) extreme points of $X^{L}$. Thus, $\forall s^{L}\in S^{L}$, we could determine the set $\tilde{\Gamma}_{t}(s^{L})$ satisfying conditions (ii) and (iii) by a finite set of belief points $W\subset X^{L}$. Given $s^{L}\in S^{L}$, the pseudocode of determining $\tilde{\Gamma}_{t}(s^{L})$ is outlined in Algorithm 5.

We initialize the set $W_{0}$ in Step 1, by including the extreme points of the belief space $X^{L}$ and at least a witness point for each $\gamma$-vector in $\Gamma_{t}(s^{L})$. These witness points are generated by the PURGE operation discussed in Section 6. We develop a concave approximation MIP in Step 2 to construct an initial set $\tilde{\Gamma}_{t}(s^{L})$ based on $W_{n}$. As the condition (ii) is only enforced on the set $W_{n}$ in Step 2, Step 3 further determines if the condition (ii) is violated on $X^{L}$. If there is an $x^{*}\in X^{L}$ at which $\tilde{v}_{t}^{L}(s^{L},x^{*})>\bar{v}_{t}^{L}(s^{L},x^{*})$, we update $W_{n}$ by including $x^{*}$. Step 4 determines $\epsilon(s^{L})$, the maximal distance between the $\tilde{v}_{t}^{L}(s^{L},x)$ and $\bar{v}_{t}^{L}(s^{L},x)$. To improve the approximation quality and reduce the gap between $\bar{v}_{t}^{L}$ and $\tilde{v}_{t}^{L}$, we also add the belief point at which the maximal distance is attained to the new set $W_{n+1}$. The program continues to update $\tilde{\Gamma}_{t}(s^{L})$ based on $W_{n+1}$. The entire procedure stops when no further improvement is identified. When it stops, the condition (ii) is guaranteed on $X^{L}$ and the maximal distance between $\bar{v}(s^{L})$ and its approximate value $\tilde{v}(s^{L})$ is bounded above by $\epsilon(s^{L})$. We now detail each step in the following subsections.

8.1 Concave Approximation on $W$

Assume $s^{L}\in S^{L}$ is given. Let $W=\{x^{i}\}$ be a set of belief points in $X^{L}$, $N=|W|$, $\tilde{z}^{i}$ be the maximum function values attained at $x^{i}$ by the set $\tilde{\Gamma}_{t}(s^{L})$, and $z^{i}$ be the function values attained at $x^{i}$ by the set $\Gamma_{t}(s^{L})$, i.e., $\tilde{z}^{i}=\tilde{v}_{t}^{L}(s^{L},x^{i})=\min\{x^{i}\gamma:\gamma\in\tilde{\Gamma}_{t}(s^{L})\}$, and $z^{i}=\bar{v}_{t}^{L}(s^{L},x^{i})=\max_{k_{1}}\min_{k_{2}}\{x^{i}\gamma^{k_{1},k_{2}}:\gamma^{k_{1},k_{2}}\in\Gamma_{t}(s^{L})\}$.

For each $\gamma^{k_{1},k_{2}}\in\Gamma_{t}(s^{L})$, define a binary variable $y^{k_{1},k_{2}}=1$ if $\gamma^{k_{1},k_{2}}\in\tilde{\Gamma}_{t}(s^{L})$ and $y^{k_{1},k_{2}}=0$ if $\gamma^{k_{1},k_{2}}\notin\tilde{\Gamma}_{t}(s^{L})$. Thus, $\tilde{\Gamma}_{t}(s^{L})=\{\gamma^{k_{1},k_{2}}:\gamma^{k_{1},k_{2}}\in\Gamma_{t}(s^{L}),y^{k_{1},k_{2}}=1\}$. Let $g$ be the distance between $\bar{v}_{t}^{L}(s^{L},x^{L})$ and $\tilde{v}_{t}^{L}(s^{L},x^{L})$ on the set $W$, that is, $g=\max_{i}z^{i}-\tilde{z}^{i}$. With the aid of additional binary variables for evaluating $\tilde{z}^{i}$, we seek the set $\tilde{\Gamma}_{t}(s^{L})$ by the following mixed integer program (2):

[TABLE]

where $M$ is a large positive number.

Minimizing the distance between $\bar{v}_{t}^{L}$ and $\tilde{v}_{t}^{L}$ (on $W$) is equivalent to minimize the maximal gap $g$. The expression $-\sum_{i\in I}\tilde{z}^{i}$ is added to the objective function in order to close the gap on $W$. The multiplier $N$ on $g$ is to ensure that the two quantities are within the same magnitude. The first to the fourth constraints compute $\tilde{z}^{i},\forall i$. Specifically, the first constraint ensures that $\tilde{z}^{i}$ is bounded above by the approximation function constructed by $\tilde{\Gamma}_{t}(s^{L})$. Each binary variable $\eta_{i}^{k_{1},k_{2}}$ is associated with a $\gamma$-vector in $\Gamma_{t}(s^{L})$ and a belief point $x^{i}$. The second and the third constraints are necessary to guarantee that $\forall x^{i}\in X^{L}$, there exists one and only one defining vector $\gamma^{k_{1},k_{2}}\in\tilde{\Gamma}_{t}(s^{L})$ such that $\tilde{z}^{i}=x^{i}\gamma^{k_{1},k_{2}}$. The fourth constraint implies that if $\gamma^{k_{1},k_{2}}$ satisfies $\tilde{z}^{i}=x^{i}\gamma^{k_{1},k_{2}}$, then $\gamma^{k_{1},k_{2}}\in\tilde{\Gamma}_{t}(s^{L})$. The fifth constraint is based on the observation that $\min_{k_{1}}\min_{k_{2}}\{x^{L}\gamma^{k_{1},k_{2}}:\gamma^{k_{1},k_{2}}\in\Gamma_{t}(s^{L})\}\leq\min_{k_{2}}\{x^{L}\gamma^{k_{1},k_{2}}:\gamma^{k_{1},k_{2}}\in\Gamma_{t}(s^{L})\},\forall k_{1}$, hence, $\tilde{\Gamma}_{t}(s^{L})\subsetneq\Gamma_{t}(s^{L})$. The second to the last constraint guarantees that $\tilde{v}_{t}^{L}(s^{L},x^{L})\leq\bar{v}_{t}^{L}(s^{L},x^{L})$ on $W$ and the last constraint determines the maximal gap between $\tilde{v}_{t}^{L}(s^{L},x^{L})$ and $\bar{v}_{t}^{L}(s^{L},x^{L})$ on $W$.

We can enhance the performance of the MIP (2) by providing a good feasible solution exploiting the structure results of $\bar{v}_{t}^{L}$. Note that for any given $k_{1}$, $\bar{v}_{t}^{k_{1}}$ computed by the set $\Gamma_{t}(s^{L},k_{1})$ is a lower bound to $\bar{v}_{t}^{L}(s^{L},x^{L})$ and satisfies all three conditions. Pick any $w\in W$. Let $z_{w}^{k_{1}}=\min\{w\gamma:\gamma\in\Gamma_{t}(s^{L},k_{1})\}$ and $k^{\prime}_{1}\in\arg\max_{k_{1}}z^{k_{1}}_{w}$. Then $\Gamma_{t}(s^{L},k^{\prime}_{1})$ is a feasible solution to the MIP (2). Determining such initial solutions is straightforward and computationally inexpensive.

8.2 Verification on $X^{L}$

The concave approximation MIP (2) only ensures that the condition (ii) is satisfied on $W\subsetneq X^{L}$. The following mixed integer program (3) further checks whether the condition is satisfied on $X^{L}$:

[TABLE]

where $M$ is a large positive number.

The objective function is to minimize the difference between the two functions for a given $s^{L}\in S^{L}$: $z^{1}(x^{L})=\max_{k_{1}}\min_{k_{2}}\{x^{L}\gamma^{k_{1},k_{2}}:\gamma^{k_{1},k_{2}}\in\Gamma_{t}(s^{L})\}$ and its approximation $z^{2}(x^{L})=\min\{x^{L}\gamma:\gamma\in\tilde{\Gamma}_{t}(s^{L})\}$. Thus, MIP (3) is the same as MIP (1) where: (i) the value $z_{2}$ is determined by the first constraint, and (ii) the second and the third constraints and the binary variable $\rho^{k_{1},k_{2}}$ associated with each vector $\gamma^{k_{1},k_{2}}\in\Gamma_{t}(s^{L})$ determine $z_{1}$.

If $\mu^{*}<0$ at the belief state $x^{*}\in X^{L}$, then $x^{*}$ should be added to $W$, and both of the MIPs (2) and (3) should be resolved. The process should continue until $\mu^{*}\geq 0$.

8.3 Approximation Error

We now determine $\epsilon(s^{L})$, the maximal difference between $z_{1}(x^{L})$ based on $\Gamma_{t}(s^{L})$ and its approximation $z_{2}(x^{L})$ based on $\tilde{\Gamma}_{t}(s^{L})$, by the following MIP (4):

[TABLE]

where $M$ is a large positive number.

The objective function is to find the maximal gap between $z_{1}(x^{L})$ and $z_{2}(x^{L})$. The first two constraints compute $z_{1}(x^{L})$ on the basis of $\Gamma_{t}(s^{L})$. As $z_{1}(x^{L})=\max_{k_{1}}v^{k_{1}}(s^{L},x^{L})$ for a given $s^{L}$, the binary variable $y^{k_{1}}$ for each $k_{1}$ and the multiple-choice constraint on $y^{k_{1}}$ ensure that there is exactly one $k_{1}$ selected to compute $z_{1}$. Meanwhile, $z_{1}(x^{L})\leq x^{L}\gamma^{k_{1},k_{2}},\forall k_{2}$ for the selected $k_{1}$. Similarly, the third and the fourth constraints compute $z_{2}(x^{L})$. The binary variable $\rho^{k}$ associated with each $\gamma$-vector in $\tilde{\Gamma}(s^{L})$ and its multiple-choice constraint guarantee that there exists one and only one $\gamma\in\tilde{\Gamma}(s^{L})$ defining $z_{2}(x^{L})$.

The approximation error is bounded above by the objective value $\epsilon(s^{L})\geq 0$, assuming at point $x^{*}\in X^{L}$. To improve the approximation quality, we also include $x^{*}$ to update $W$ and $\tilde{\Gamma}(s^{L})$. Let $2^{nd}best$ be the second best leader’s action for $\min_{k_{1}}\sum_{x^{i}\in W}|\bar{v}(s^{L},x^{i})-\bar{v}^{k_{1}}(s^{L},x^{i})|$. The following Proposition shows that this procedure guarantees $\epsilon(s^{L})\leq\max_{x\in X^{L}}\max_{k_{1}}|\bar{v}^{k_{1}}(s^{L},x^{L})-\bar{v}^{2^{nd}best}(s^{L},x^{L})|$. That is, the approximation function at any belief point is no worse than the performance induced by the leader’s second best action (on W). Moreover, the approximation error of the proposed approach could be zero when there is a dominant action of the leader.

Proposition 6**.**

$\epsilon(s^{L})\leq\max_{x\in X^{L}}\max_{k_{1}}|\bar{v}^{k_{1}}(s^{L},x^{L})-\bar{v}^{2^{nd}best}(s^{L},x^{L})|$. Furthermore, if there is a leader’s action $a^{L,*}$ such that $G(s^{L},a^{L,*})$ pairwisely dominates $G(s^{L},a^{L}),\forall a^{L}\in A^{L},a^{L}\neq a^{L,*}$, then $\epsilon(s^{L})=0$.

Proof.

The first result follows from that (i) $\bar{v}^{2^{nd}best}(s^{L},x^{L})$ is a feasible solution to the MIP (2); (ii) it is a second best minimizer of $\min_{\bar{v}^{k_{1}}}\sum_{i\in W}[\bar{v}(s^{L},w_{i})-\bar{v}^{k_{1}}(s^{L},w_{i})]$; and (iii) the construction of the set $W$ (Step 4 in Algorithm 5). The second result follows as the set $G(s^{L},a^{L,*})$ satisfies the conditions (i)-(iii) and the pairwise dominance assumption implies $\tilde{\Gamma}_{t}(s^{L})=\Gamma_{t}(s^{L})=G(s^{L},a^{L,*})$. ∎∎

Furthermore, let $T:V\rightarrow V$ be the (nonlinear) operator such that $\forall u\in V$, $Tu$ is the approximation of $u$ satisfying the conditions (i)-(iii), and $\tilde{v}_{T}^{L}=Tv^{L}_{T}$. At each iteration, Algorithm 1 evaluates $\bar{v}_{t}^{L}=H\tilde{v}^{L}_{t+1}$ and approximates $\bar{v}_{t}^{L}$ by $\tilde{v}_{t}^{L}=T\bar{v}^{L}_{t}$. Thus, $\bar{v}_{t}^{L}=(H\circ T)\bar{v}_{t+1}^{L}.$

Proposition 7**.**

$\tilde{v}^{L}_{t}\leq\bar{v}^{L}_{t}\leq v_{t}^{L}$* and $||\bar{v}_{t}^{L}-\bar{v}_{t-1}^{L}||\leq\beta^{T-t}||\bar{v}_{T}^{L}-\bar{v}_{T-1}^{L}||$.*

Proof.

$v_{t}^{L}\geq\bar{v}_{t}^{L}\geq\tilde{v}_{t}^{L}$ is obvious by the definition of $T$ and the fact that if $u\leq v,u,v\in V$, then $Hu\leq Hv$. For the second part, note if $||u-v||=\epsilon$, then $u-\epsilon\leq v\leq u+\epsilon$. By the definition of $T$, $Tu-\epsilon\leq Tv\leq v\leq u+\epsilon$. Similarly, we also have $Tv-\epsilon\leq Tu\leq u\leq v+\epsilon$. Thus, $||Tu-Tv||\leq||u-v||$. Now, $||\bar{v}_{t}^{L}-\bar{v}_{t-1}^{L}||=||(H\circ T)\bar{v}_{t+1}^{L}-(H\circ T)\bar{v}_{t}^{L}||\leq\beta||T\bar{v}_{t+1}^{L}-T\bar{v}_{t}^{L}||\leq\beta||\bar{v}_{t+1}^{L}-\bar{v}_{t}^{L}||\leq...\leq\beta^{T-t}||\bar{v}_{T}^{L}-\bar{v}_{T-1}^{L}||$. ∎∎

Thus, if we solve the finite horizon problem for a larger and larger $T$, the lower bound function $\bar{v}_{t}^{L}$ will also converge (and will be a lower bound of the fixed point $v^{*}=Hv^{*}$). Although we only focus on the finite horizon case in this paper, this result shows that the developed algorithm can also be used to obtain, at least approximately, a lower bound of leader’s value function in the infinite planning horizon. We remark that there is no algorithm for the infinite-horizon general-sum POSG in the literature yet.

9 A Security Application

In this section, we describe a class of dynamic resource allocation problems in security context, where the developed model and solution procedure can be employed.

Consider a defender with limited defensive resources is protecting a set of critical nodes against an intelligent adversary over time. A node could be a manufacturing/factory site, a computer on a network, or a security checkpoint in an airport. At any time, an adversary can choose to breach the security of any node at certain levels. The objective of the defender is to minimize the number of breaches and losses generated by these violations. However, it can be difficult to quantify the reward structure of the adversary, its value over each node, and its rationality. Moreover, due to limited resources and capabilities, the defender may not be fully aware of the attacker state (e.g., exact locations, attack capabilities); but the defender may infer the attacker’s state through reported locations, historical attack records, screening, sensors and detectors, and unstructured text data from social media. On the other hand, the defender’s state is also only partially observable to the adversary in many realistic scenarios (e.g., some defensive resources can be camouflaged). While attacking the system, the adversary may adjust its behavior and target based on its updated information on the defensive resource allocation. In order to provide decision support to the defender, we could model such problems as a leader-follower POSG under the worst-case scenario, where the defender is the leader and the adversary is the follower. Furthermore, our solution procedure can determine dynamic defense policies that timely adjust resource allocation on the basis of all available real-time information in order to protect the system.

Dynamic defense policies are a class of defense policies aligned with the trending philosophy of “Moving Target Defense” (MTD) (Miehling et al. 2015). Currently, the static configuration and operation of a system has presented adversaries with an incredible advantage as adversaries can take their time to study the system and plan attacks (Department of Homeland Security 2015). The focus of MTD is to dynamically change a system in order to shift or reduce the attack surface that can be exploited by adversaries to attack the system (Zhuang et al. 2014).

Many concrete examples in the dynamic risk assessment and security game literature are in this application area, including cybersecurity, traveling inspection (Ahmadi et al. 2018; Bakir and Kardes 2009; Haskell et al. 2014; Kardes 2014; Lopez et al. 2013; Poolsappasit et al. 2012; Shameli-Sendi et al. 2012; Yang et al. 2014; Wang et al. 2019). Here, we use the liquid egg production problem presented in Zhang (2013) for illustration. For the completeness of the paper, we briefly restate the security problem in Section 9.1.

9.1 Problem Description

Liquid egg products are widely used by the food service industry and as ingredients in other food products such as mayonnaise and ice cream (United States Department of Agriculture Food Safety and Inspection Service 2015). A deliberate contamination in the liquid egg products by an adversary will breach food safety, leading to excessive morbidity and mortality. Zhang (2013) identified the critical components of a liquid egg production process, including collecting vats, raw material tanks, pasteurization, and finished product tanks (Fig. 6). An unknown adversary may use this system as a toxin delivery vehicle by inserting a toxin (e.g., botulinum) at these components (“targets”). The consequence of such attacks occurred at each component is defined as the number of contaminated packages, and the numerical values of the consequence have been analyzed in literature.

We now illustrate how to use the developed method to support the production manager with limited resources in selecting a sequence of actions for protecting the system against an unknown adversary, in order to maximize the long-run productivity of the production facility. We allow for multiple attacks and each attack can be successful or unsuccessful. An unsuccessful attack occurs when the adversary launches an attack but fails to insert any toxin to the system (e.g., the adversary is caught by the manager during the attack). Thus, the production process will not be affected, and the manager needs to prepare for next possible attacks. After a successful attack, however, the manager has to stop the production process to remove inserted toxin and clean up the system. Thus, the game stops whenever a successful attack occurs. As the pasteurization process can significantly reduce the effectiveness of the botulinum toxin, we assume the manager needs to protect three targets: Collecting Vat (Target 1), Raw Production Tank (Target 2), and the Finished Product Tank (Target 3).

State spaces, action spaces, and system dynamics: We assume the manager can only protect one target each time (e.g., visit and inspect one critical component each time). Thus, the state of the manager is the target under protection. The state space of the manager is $S^{L}=\{\mbox{Target 1 Protected},\mbox{Target 2 Protected},\mbox{Target 3 Protected},\\ \mbox{Attacked}\}$, where the “Attacked” state indicates toxin has been successfully inserted to the system. The manager decides which target to protect dynamically based on its own information data.

The state of the adversary is the location of the adversary. Hence, $S^{F}$= {Target 1, Target 2, Target 3, Attacked}, and $|S|=|S^{F}||S^{L}|=16$. At Target $i$, the adversary can either attack the target or switch to another target. Thus, there are 3 actions for each agent (9 action pairs) in each state.

The system transits to a new state once both the agents have selected actions.

Observation space: The manager’s observations of the adversary are the possible locations of the adversary. Thus, $Z^{L}=S^{F}$, $|Z^{L}|=4$. We assume that the manager has the ability to detect an attack (e.g., by testing) if the attack has successfully occurred. Specifically, the observation matrix is $P(z^{L}|s^{F})=\epsilon_{s^{F},z^{L}}$, where $0\leq\epsilon_{s^{F},z^{L}}\leq 1$ and $\sum_{z^{L}\in Z^{L}}\epsilon_{s^{F},z^{L}}=1$.

Reward structure, criterion, and objective: The system can produce $L$ number of qualified packages under normal operations. A successful attack with 2000 grams of botulinum at location $i$ can result in $L^{i}_{c}$ number of contaminated packages, $i\in$ {Target 1, Target 2, Target 3}. The numerical values of $L$ and $L^{i}_{c}$ are estimated by the simulation model developed in Zhang (2013). If there is no attack, the reward of the manger $r^{L}(s,a),s^{L}\neq\mbox{``Attacked"}$ is its normal productivity $L$. If a successful attack has been detected, no package will be produced because the manager has to stop the production and clean up the system. We assume the manager will receive an additional bonus $b>0$ for successfully preventing an attack. Let $p$ ($q$) be the probability of having a successful attack at a protected (unprotected) target. Assume $0\leq p<<q\leq 1$. The cleanup cost is $C$ for the manager to remove toxin from the system after a successful attack. Thus,

[TABLE]

The criterion of the manager is the expected finite horizon total discounted reward $v^{L}(\zeta_{0}^{L})=E\{\sum_{t=0}^{T}\beta^{t}r^{L}(s_{t},a_{t})|\zeta_{0}^{L}\}$, where we assume $\beta=0.85$ and $T=30$ for illustrative purpose. The objective of the manager is to maximize the value of criterion under the worst-case scenario.

9.2 Numerical Results

We first use $t=T$ to illustrate the procedure in Algorithm 1. Table 1 summarizes the $\gamma$-vectors after the PURGE and DOMINANCE operations for $s^{L}\neq$“Attacked” at $t=T$. Thus, $v_{T}^{L}(s^{L},x^{L})=\max_{k_{1}}\min_{k_{2}}\{x^{L}\gamma^{k_{1},k_{2}}:\gamma\in\Gamma_{T}(s^{L})\}$. Fig. 1 shows the graph of the true value function $v_{T}^{L}$ and its approximation $\tilde{v}_{T}^{L}$ projected on the non-absorbing states of the follower (i.e., $s^{F}\neq$“Attacked”). Clearly, $v_{T}^{L}(s^{L},x^{L})$ (in blue) is not a concave function and $\tilde{v}_{T}^{L}(s^{L},x)$ (in red) is indeed the best piecewise linear concave approximation function of $v_{T}^{L}(s^{L},x)$. Let $P\subset X^{L}$ be the region where the approximation is accurate, i.e., $P=\{x\in X^{L}:v_{T}^{L}(s^{L},x^{L})=\tilde{v}_{T}^{L}(s^{L},x^{L})\}$. Then $|P|/|X^{L}|=78.63\%$ (in terms of the Lebesgue measure), and the maximal approximation error ($4.36\%$) occurs around the extreme point $e_{2}$.

Fig. 7 shows the convergence result of the overall procedure. The maximum deviation of the value function $\bar{v}_{t+1}^{L}$ from $\bar{v}_{t}^{L}$

[TABLE]

declines as the algorithm proceeds, and the value function $\bar{v}^{L}$ converges to $\bar{v}^{L,*}$ after 27 iterations.

The entire solution procedure was performed on an Intel 3.10 GHz processor having 6.00 GB memory. The total computation time for $T=30$ was 156.31 seconds, where the PURGE, DOMINANCE, and APPROXIMATION operations accounted for 4.78%, 11.86%, and 83.45%, respectively. As at least a witness point was associated with each $\gamma$-vector in the concave approximation MIP (2), the sizes of MIPs in the APPROXIMATION step are significantly larger than those of the MIPs in the PURGE step and DOMINANCE step (could be 10$\sim$20 times larger). We remark that the dynamic programming approach for POSGs in Hansen et al.(2004) quickly ran out of memory after horizon $T=4$ for a small two-agent problem (the sizes of state, observation, and action spaces for each agent are all two).

To illustrate our results and support the validation of the solution procedure and computational results, we consider the following three scenarios:

1. (i)

Scenario 1: The policy for the leader is constructed according to Algorithm 1, and the follower selects the action that minimizes the leader’s criterion value; 2. (ii)

Scenario 2: The policy for the leader is constructed according to Algorithm 1, and the follower randomly selects its action; 3. (iii)

Scenario 3: The leader protects the most vulnerable target (which is the target that generates the largest negative consequence if attacked), and the follower selects the action that minimizes the leader’s criterion value.

We assume the measure of performance is the leader’s total discounted reward for each sample path and present in Fig. 8 the distribution of the performance measure for each scenario, based on 1000 simulations. We note the following from Fig. 8:

1. (i)

Scenario 1 has a lower mean than Scenario 2, which illustrates that if the policy for the leader is constructed according to Algorithm 1 and the follower uses policies other than selecting actions to minimize the leader’s criterion value (e.g., irrational), then the performance of the leader will only improve. That is, the proposed algorithm protects the leader against all possible decision making processes of the adversary. 2. (ii)

Scenario 1 has a higher mean than Scenario 3, which illustrates that if the leader uses policies other than the policy constructed according to Algorithm 1 and the follower selects actions that minimize the leader’s criterion, the performance of the leader will degrade. It also shows that constantly protecting the most vulnerable target is not always the best option to the leader in the presence of an intelligent adversary. This is because in a dynamic environment, the adversary will learn the leader’s strategy and thus attacks other targets accordingly. It further illustrates the importance of developing dynamic defensive strategies and defensible systems for protecting against intelligent and adaptive adversaries.

10 Conclusions

This paper introduced a worst-case analysis to a general-sum partially observable stochastic game with two non-cooperative agents, a leader and a follower, where the leader has little knowledge of the follower. A general-sum partially observable stochastic game can be transformed to a simpler single-agent problem under the worst-case scenario; however, it cannot be transformed to a standard POMDP. While the worst-case modeling can be viewed as a POMDP with imprecise parameters, there is no as yet established algorithms for these problems in the literature. In order to determine a baseline performance for the leader, we showed the optimal value function of the leader can be non-convex and developed a recursive algorithm to construct a lower bound of the leader’s value function in the finite horizon and its associated policy. We further analyzed the quality of the lower bound and showed that the proposed procedure can be used to approximately evaluate the leader’s performance in the infinite horizon case. The use of the model and the solution procedure was illustrated by a liquid egg production example in a security context.

The lower bound was constructed by the sets $\tilde{\Gamma}_{t}(s^{L})\subset\Gamma_{t}(s^{L}),\forall s^{L}\in S^{L}$. Future research should further improve the lower bound by efficiently searching $\gamma\notin\Gamma_{t}(s^{L})$. Another research direction would be a detailed discussion for the infinite planning horizon problem. Moreover, the developed analysis provided a benchmark result for an agent in a dynamic, multi-agent partially observable stochastic environment. Thus, the follow-up research may include investigating the value of improved understanding of the adversarial behaviors by comparing with these benchmark results.