A dynamic game approach to distributionally robust safety specifications for stochastic systems

TL;DR

This paper introduces a distributionally robust safety control method for stochastic systems, using a dynamic game approach to handle uncertainty in disturbance distributions and ensuring safety with limited distributional information.

Contribution

It develops a novel dynamic game formulation for distributionally robust safety policies and provides a practical duality-based solution to infinite-dimensional minimax problems.

Findings

The proposed method is robust against distributional errors in disturbances.

It transforms complex minimax problems into solvable semi-infinite programs.

Numerical tests confirm improved safety robustness over standard tools.

Abstract

This paper presents a new safety specification method that is robust against errors in the probability distribution of disturbances. Our proposed distributionally robust safe policy maximizes the probability of a system remaining in a desired set for all times, subject to the worst possible disturbance distribution in an ambiguity set. We propose a dynamic game formulation of constructing such policies and identify conditions under which a non-randomized Markov policy is optimal. Based on this existence result, we develop a practical design approach to safety-oriented stochastic controllers with limited information about disturbance distributions. This control method can be used to minimize another cost function while ensuring safety in a probabilistic way. However, an associated Bellman equation involves infinite-dimensional minimax optimization problems since the disturbance distribution may have a continuous density. To resolve computational issues, we propose a duality-based reformulation method that converts the infinite-dimensional minimax problem into a semi-infinite program that can be solved using existing convergent algorithms. We prove that there is no duality gap, and that this approach thus preserves optimality. The results of numerical tests confirm that the proposed method is robust against distributional errors in disturbances, while a standard stochastic safety specification tool is not.

Equations

$$x_{t+1}=f(x_t,u_t,w_t)t\in \mathcal{T},x_0=\mathbf{x},$$

$$P_{\mathbf{x}}^{\text{mbox}\text{safe}}(\pi ,\gamma ;A):={\mathbb{P}}^{\pi ,\gamma }\{x_t\in A\forall t\in \bar{\mathcal{T}},x_0=\mathbf{x}\},$$

$$S_{\alpha }(\pi ,\gamma ;A):=\{\mathbf{x}\in {\mathbb{R}}^n|P_{\mathbf{x}}^{\text{mbox}\text{safe}}(\pi ,\gamma ;A)\ge \alpha \}.$$

$$\underset{\gamma \in \Gamma }{\inf }{P}_{\mathbf{x}}^{\text{mbox}\text{safe}}({\pi }^{\star },\gamma ;A)\ge \underset{{\gamma }^{\prime }\in \Gamma }{\inf }{P}_{\mathbf{x}}^{\text{mbox}\text{safe}}(\pi ,{\gamma }^{\prime };A)\forall \pi \in \Pi .$$

$$S_{\alpha }^{\star }(A)=\{\mathbf{x}\in {\mathbb{R}}^n|\underset{\pi \in \Pi }{\sup }\underset{\gamma \in \Gamma }{\inf }{P}_{\mathbf{x}}^{\text{mbox}\text{safe}}(\pi ,\gamma ;A)\ge \alpha \}.$$

$$P_{\bm{x}}^{\mbox{\tiny safe}}(\pi,\gamma;A)=\mathbb{E}^{\pi,\gamma}\bigg{[}\prod_{t=0}^{T}\mathbb{1}_{A}(x_{t})\bigg{]},$$

$$\underset{\pi \in \Pi }{\sup }\underset{\gamma \in \Gamma }{\inf }{P}_{\mathbf{x}}^{\text{mbox}safe}(\pi ,\gamma ;A).$$

$${\hat{g}}_t(\mathbf{x},\mathbf{u},\mathbf{\mu }):={\int }_{{\mathbb{R}}^n}{g}_t(\xi )Q_t(d\xi |\mathbf{x},\mathbf{u},\mathbf{\mu })$$

$${\mathbb{T}}_{t}v(\mathbf{x},\mathbf{y}):=\underset{\mathbf{u}\in \mathbb{U}(\mathbf{x})}{\sup }\underset{\mathbf{\mu }\in {\mathbb{D}}_t}{\inf }{1}_A(\mathbf{x}){\int }_{{\mathbb{R}}^n}v(\xi )Q_t(d\xi |\mathbf{x},\mathbf{u},\mathbf{\mu }).$$

$${\mathbb{T}}_{t}v(\mathbf{x})=\underset{\mathbf{\mu }\in {\mathbb{D}}_t}{\inf }\left[1_A(\mathbf{x}){\int }_{{\mathbb{R}}^n}v(\xi )Q_t(d\xi |\mathbf{x},\kappa (\mathbf{x}),\mathbf{\mu })\right].$$

$$\hat{v}(\mathbf{x},\mathbf{u},\mathbf{\mu }):={\int }_{{\mathbb{R}}^n}v(\xi )Q_t(d\xi |\mathbf{x},\mathbf{u},\mathbf{\mu }).$$

$$\begin{array}{rl}& \lim \underset{j\to \infty }{\sup }{\int }_{{\mathbb{R}}^n}v(\xi )Q_t(d\xi |{\mathbf{x}}_j,{\mathbf{u}}_j,{\mathbf{\mu }}_j)\le {\int }_{{\mathbb{R}}^n}{v}_k(\xi )Q_t(d\xi |\mathbf{x},\mathbf{u},\mathbf{\mu })\end{array}$$

$$\begin{array}{rl}|1_A(\mathbf{x})\hat{v}(\mathbf{x},\mathbf{u},\mathbf{\mu })|& =1_A(\mathbf{x}){\int }_{{\mathbb{R}}^n}|v(\xi )|Q_t(d\xi |\mathbf{x},\mathbf{u},\mathbf{\mu })\\ & \le \Vert v{\Vert }_{\infty }<\infty .\end{array}$$

$$v_t(\mathbf{x}):={\mathbb{T}}_t\circ {\mathbb{T}}_{t+1}\circ \cdots \circ {\mathbb{T}}_{T-1}{1}_A(\mathbf{x}).$$

$$v_t(\mathbf{x})=\underset{\mathbf{\mu }\in {\mathbb{D}}_t}{\inf }\left[1_A(\mathbf{x}){\int }_{{\mathbb{R}}^n}{v}_{t+1}(\xi )Q_t(d\xi |\mathbf{x},{\varphi }_t(\mathbf{x}),\mathbf{\mu })\right]$$

$$v_0(\mathbf{x})=\underset{\gamma \in \Gamma }{\inf }{P}_{\mathbf{x}}^{\text{mboxsafe}}({\pi }^{\star },\gamma ;A).$$

$$v_T(\mathbf{x})=1_A(\mathbf{x}),$$

$$\begin{array}{rl}& v_t(\mathbf{x})=\underset{\mathbf{u}\in \mathbb{U}(\mathbf{x})}{\max }\underset{\mathbf{\mu }\in {\mathbb{D}}_t}{\inf }{1}_A(\mathbf{x}){\int }_{{\mathbb{R}}^l}{v}_{t+1}(f(\mathbf{x},\mathbf{u},\mathbf{w}))d\mathbf{\mu }(\mathbf{w}).\end{array}$$

$$\begin{array}{rl}& {\varphi }_t^{\text{mbox}safe}(x_t)\in \arg \underset{\mathbf{u}\in \mathbb{U}(x_t)}{\max }\underset{\mathbf{\mu }\in {\mathbb{D}}_t}{\inf }{1}_A(x_t){\int }_{{\mathbb{R}}^l}{v}_{t+1}(f(x_t,\mathbf{u},\mathbf{w}))d\mathbf{\mu }(\mathbf{w}).\end{array}$$

$$S_{\alpha }^{\star }(A)=\{\mathbf{x}\in {\mathbb{R}}^n|v_0(\mathbf{x})\ge \alpha \}$$

$$\begin{array}{rl}{S}_{\alpha ,t}^{\star }(A):=\{& \mathbf{x}\in {\mathbb{R}}^n|\forall \gamma \in \Gamma \exists \pi \in \Pi \text{mbox}s.t.{\mathbb{P}}^{\pi ,\gamma }(x_s\in A\forall s=t,\cdots ,T,x_t=\mathbf{x})\ge \alpha \}.\end{array}$$

$$S_{\alpha ,t}^{\star }(A)=\{\mathbf{x}\in {\mathbb{R}}^n|v_t(\mathbf{x})\ge \alpha \}$$

$$u_{t}^{\tiny\mbox{safe}}\left\{\begin{array}[]{ll}\in\mathbb{U}(x_{t})&\mbox{ if }x_{t}\in\{\bm{x}\>|\>f(\bm{x},\bm{u},\bm{w})\in S_{\alpha,t+1}^{\star}(A)\;\forall\bm{u}\in\mathbb{U}(\bm{x})\;\forall\bm{w}\in W_{t}\}\\ =\phi_{t}^{\tiny\mbox{safe}}(x_{t})&\mbox{ otherwise}.\end{array}\right.$$

$$x_t\in S_{\alpha ,t}^{\star }(A)\forall t\in \bar{\mathcal{T}},$$

$$\begin{array}{rl}{\mathbb{P}}^{{\pi }^{\text{mbox}safe},\gamma }(x_s\in A\forall s\in {\mathcal{T}}_{t+1})& \ge {\mathbb{P}}^{{\pi }^{\text{mbox}safe},\gamma }(x_s\in A\forall s\in {\mathcal{T}}_t)\\ & \ge \alpha \forall \gamma \in \Gamma ,\end{array}$$

$$\underset{\pi \in \Pi }{\inf }\underset{\gamma \in \Gamma }{\sup }{\mathbb{E}}^{\pi ,\gamma }\left[\sum _{t=0}^{T-1}r(x_t,u_t)+q(x_T)\right]$$

$$\begin{array}{rl}{\mathbb{D}}_t:=\{& {\mu }_t\in M_{+}(W_t)|{\mu }_t(W_t)=1,\\ & |{\mathbb{E}}_{{\mu }_t}[w_t]-m_t|\le b_t,\\ & {\mathbb{E}}_{{\mu }_t}[(w_t-m_t)(w_t-m_t{)}^{\top }]⪯c_t{\Sigma }_t\},\end{array}$$

$$\mathbf{P}:\underset{\mathbf{\mu }}{\inf }$$

$$m_t-b_t\le {\int }_{W_t}\mathbf{w}d\mathbf{\mu }(\mathbf{w})\le m_t+b_t$$

$${\int }_{W_t}(\mathbf{w}-m_t)(\mathbf{w}-m_t{)}^{\top }d\mathbf{\mu }(\mathbf{w})⪯c_t{\Sigma }_t$$

$${\int }_{W_t}d\mathbf{\mu }(\mathbf{w})=1$$

$$\mathbf{\mu }\in M_{+}(W_t).$$

Full text

Key words. Distributionally robust optimization, safety specifications, reachability analysis, distributional ambiguity, stochastic systems, dynamic games, duality.

1 Introduction

Various critical decision-making and control problems associated with engineering and socio-technical systems are subject to uncertainties. Large-scale data collected from the Internet of Things and cyber-physical systems can provide information about the probability distribution of these uncertainties. Statistical learning and filtering methods also support the construction of data-driven distribution models of uncertainty based on the observed data. Such distributional information can be used to dramatically improve the performance of closed-loop systems if they adopt appropriate controllers, that reduce the conservativeness of classical techniques, such as robust control. Several concerns have been raised about how best to incorporate the collected data into critical control and decision-making problems. These concerns center on safety, risk, robustness and reliability because the data and statistical models extracted from the data often result in inaccurate distributional information. Among them, we focus on the safety issue and develop a safety specification and management tool with ambiguous information about the probability distribution of disturbances.

For safety-critical systems subject to uncertain disturbances, reachability-based safety specification techniques have been used to compute the reachable sets and safe sets, which allow one to verify that a system is evolving within a safe range of operation and to synthesize controllers to satisfy safety constraints (e.g., [5], [47], [28], [7], [23], [36], [17], [40], [2], [29], [16], [8]). These methods assume that disturbances lie in a compact set, and thus require information only about the support of disturbances. However, these techniques often produce conservative results as no additional information about the probability distribution of uncertain disturbances is used. These deterministic methods are a natural choice when the data of disturbances are not continuously collected, and thus a reliable stochastic model is not available for them. Advances in sensing, communication, and computing technologies as well as statistical learning and estimation tools make it possible to shift this paradigm; sensors, data storage and computing infrastructure in various systems can now provide data to help estimate the probability distribution of disturbances. Stochastic reachability analysis tools are based on the assumption that the full probability distribution of disturbances is available and can often be used to reduce the conservativeness of deterministic safe set computations. However, this assumption is often restrictive in practice because obtaining an accurate distribution requires large-scale high-resolution sensor measurements over a long training period or multiple periods. Furthermore, the accuracy of the distribution obtained with computational methods is often unreliable as it is subject to the quality of the observations, statistical learning or filtering methods, and prior knowledge about the disturbances. Thus, probabilistic safety specification tools can be misleading and lead to the design of an unreliable controller that may violate safety constraints when distributional information about disturbances is inaccurate.

Fig. 1 illustrates the tradeoff between required information and conservativeness in deterministic and stochastic safety specification methods. This study aims to bridge the gap between the two methods by proposing a distributionally robust safety specification tool. Our approach assumes that the distribution of disturbances is not fully known but lies in a so-called ambiguity set of probability distributions. If, for example, only the support, mean, and variance of an uncertain variable are reliably estimated, the ambiguity set can be chosen so as to contain distributions consistent with the empirical estimates. The proposed distrbutionally robust safe policy maximizes the probability of a system remaining within a desired set for all times subject to the worst possible disturbance distribution in the ambiguity set. Therefore, the probabilistic safe set of the closed-loop system is robust against distributional errors within the ambiguity set.

This paper proposes a dynamic game formulation of constructing distributionally robust safe policies and safe sets. Specifically, it is a two-player zero-sum dynamic game in which Player I selects a policy by which the controller can maximize the probability of safety, while (fictitious) Player II adversarially determines a strategy for the probability distribution of disturbances to minimize the same probability. Player II’s action space is generally infinite dimensional since the disturbances may have a continuous density function. Therefore, the Bellman equation for this dynamic game problem involves infinite-dimensional optimization problems that are computationally challenging. Furthermore, the existence of a distributionally robust safe policy is not guaranteed.

The contributions of this work are threefold. First, we characterize conditions under which a non-randomized Markov policy is optimal for Player I (controller). This characterization helps greatly reduce the control strategy space we need to search for because it is enough to restrict our attention to non-randomized Markov policies. Furthermore, the existence of a non-randomized Markov policy guarantees that the outer maximization problem (for Player I) of an associated Bellman equation is solvable—that is, it is feasible and has an optimal solution. Second, we develop a design approach to a safety-oriented stochastic controller with limited information about disturbance distributions. This control method can be used to minimize another cost function while guaranteeing that the probability for a system being safe for all remaining stages is greater than or equal to a pre-specified threshold, regardless of how the disturbance distribution is chosen in an ambiguity set. Third, we propose a duality-based reformulation method for the Bellman equation in cases with moment uncertainty. We show that there is no duality gap in the inner minimization problem (for Player II) of the Bellman equation, which is an infinite-dimensional optimization problem. Using the strong duality result, we reformulate each infinite-dimensional minimax problem in the Bellman equation as a semi-infinite program without sacrificing optimality. This reformulation alleviates the computational issue arising from the infinite dimensionality of the original Bellman equation because the reformulated Bellman equation can be solved through the use of existing convergent algorithms for semi-infinite programs.

1.1 Related Work

A probabilistic reachability tool for stochastic differential equations with jumps has been proposed; it uses a Markov chain approximation to propagate the transition probabilities of the Markov chain backward in time starting from a target set [21], [38], [39]. In [37], barrier certificates are employed to calculate an upper bound of the probability that a system will reach a target set. Additionally, [31] proposes a toolbox that supports expectation-based reachability problems associated with a class of continuous-time stochastic (hybrid) systems by extending the celebrated Hamilton–Jacobi–Isaacs reachability analysis [48], [30]. A partial differential equation characterization of continuous-time stochastic reach-avoid problems is studied in [32] based on the theory of discontinuous viscosity solutions. For discrete-time stochastic hybrid systems, an elegant dynamic programming approach has been proposed to compute the maximal probability of safety [1]. This method has been extended to stochastic reach–avoid problems [45], stochastic hybrid games [10], and partially observable stochastic hybrid systems [25], [26]. However, all the aforementioned methods are based on the possibly restrictive assumption that the probability distribution of disturbances is completely known.

This work also closely relates to distributionally robust control, which is an emerging stochastic control method. This method is based on single-stage distributionally robust stochastic optimization that minimizes the worst-case cost, assuming that the probability distribution of uncertain variables lies within an ambiguity set of distributions (e.g., [43], [12], [13], [6], [9], [50]). For multi-stage problems, a distributionally robust Markov decision process (MDP) formulation has recently been developed while focusing on finite-state, finite-action MDPs [51], [53]. For cases with moment uncertainty, [49] investigates linear feedback strategies in linear-quadratic settings with risk constraints and proposes a semidefinite programming approach. We extend the theory of distributionally robust control to the case of continuous state spaces and apply it to reachability analysis and safety specifications.

1.2 Organization

The remainder of this paper is organized as follows. In Section 2, we define distributionally robust safe sets and policies and introduce a dynamic game formulation to construct them. Section 3 contains a dynamic programming solution to the problem of computing optimal policies. In particular, we characterize conditions under which a Markov control policy is optimal, and propose a safety-oriented controller design method. In Section 4, we consider ambiguity sets with moment uncertainty and show that an associated Bellman equation can be formulated as a semi-infinite program. An application of the proposed safety specification tool is illustrated through examples in Section 5.

1.3 Notation

Given a Borel space $X$, $\mathcal{B}(X)$ represents its Borel $\sigma$-algebra. Given Borel spaces $X$ and $Y$, $Q(A|y)$ denotes a transition probability (or a stochastic kernel) from $Y$ to $X$, where $Q(\cdot|y)$ is a probability measure on $\mathcal{B}(X)$ for each $y\in Y$ and $Q(A|\cdot)$ is a measurable function on $Y$ for each $A\in\mathcal{B}(X)$. Also, $M(X)$ denotes the Banach space of finite signed measures on $\mathcal{B}(X)$ and $M_{+}(X)$ represents its positive cone. For simplicity, we use the following notation of time indexes: $\mathcal{T}:=\{0,1,\cdots,T-1\}$ and $\bar{\mathcal{T}}:=\{0,1,\cdots,T\}$.

2 Distributionally Robust Safe Sets and Policies

2.1 Stochastic Systems with Distributional Ambiguity

Consider a discrete-time stochastic system of the form

[TABLE]

where $x_{t}\in\mathbb{R}^{n}$ is the state, $u_{t}\in\mathbb{R}^{m}$ is the control input, $w_{t}\in\mathbb{R}^{l}$ is the stochastic disturbance, and $f:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}^{l}\to\mathbb{R}^{n}$ is a measurable function. We assume that the disturbance process $\{w_{t}\}_{t=0}^{T-1}$ is defined on a probability space $(\Omega,\mathcal{F},\mathbb{P})$, and that $w_{s}$ and $w_{t}$ are independent for any $s\neq t$. As mentioned in Section 1, it is often difficult to obtain full information about the probability distribution $\mu_{t}$ of $w_{t}$. In many cases, only estimates of its support, mean, and variance are available. Furthermore, such estimates are rarely accurate. This issue of imperfect disturbance distributions degrades the practicality of classical stochastic reachability tools. To resolve this problem, we allow errors in such distributional information and quantify the probability of safety with the worst-case disturbance distribution. To mathematically model distributional ambiguity, we assume that the true probability distribution $\mu_{t}$ of $w_{t}$ is contained in a so-called ambiguity set of distributions, denoted by $\mathbb{D}_{t}$. Note that $\mathbb{D}_{t}$ is a subset of the space $M_{+}(\mathbb{R}^{l})$ of signed measures and thus is infinite dimensional. An example of such ambiguity sets can be found in Section 4.1. We assume that the set $\mathbb{D}_{t}$ of distributions is not empty for each $t$.

We now briefly discuss admissible control and disturbance distribution strategies. Let $H_{t}$ be the set of histories up to stage $t$, whose element takes the form $h_{t}=(x_{0},u_{0},\mu_{0},\cdots,x_{t-1},u_{t-1},\mu_{t-1},x_{t})$. The set of admissible control strategies is chosen as $\Pi:=\{\pi=(\pi_{0},\cdots,\pi_{T-1})\>|\>\pi_{t}(\mathbb{U}(x_{t})|h_{t})=1\;\forall h_{t}\in H_{t}\}$, where $\pi_{t}$ is a stochastic kernel from $H_{t}$ to $\mathbb{R}^{m}$ and $\mathbb{U}(x_{t})$ is the set of admissible actions given state $x_{t}$. Note that this strategy space is sufficiently broad to contain randomized non-Markov policies. We assume that there exists a measurable function $\pi:\mathbb{R}^{n}\to\mathbb{R}^{m}$ such that $\pi(\bm{x})\in\mathbb{U}(\bm{x})$ for all $\bm{x}\in\mathbb{R}^{n}$. By viewing the disturbance as an adversarial player who chooses the disturbance’s probability distribution based on the available information, the set of admissible disturbance distribution strategies is similarly defined as $\Gamma:=\{\gamma=(\gamma_{0},\cdots,\gamma_{T-1})|\gamma_{t}(\mathbb{D}_{t}|h_{t}^{e})=1\;\forall h_{t}^{e}\in H_{t}^{e}\}$, where $H_{t}^{e}$ is an extended set of histories up to stage $t$, whose element is of the form $h_{t}^{e}=(x_{0},u_{0},\mu_{0},\cdots,x_{t-1},u_{t-1},\mu_{t-1},x_{t},u_{t})$. Note that the distributional constraints in the ambiguity set $\mathbb{D}_{t}$ is encoded in this strategy space. The disturbance (or the adversarial player) can use slightly more information than the controller; the disturbance is aware of the controller’s action at stage $t$, $u_{t}$, in addition to the history $h_{t}$. However, the controller cannot be aware of the disturbance distribution’s realization $\mu_{t}$ when making a decision at stage $t$.

2.2 Distributionally Robust Safety Specifications

Our goal is to develop a probabilistic safety specification tool that is robust against errors in the probability distribution of the disturbance $\{w_{t}\}$. Specifically, we compute the worst-case probability of a system remaining in a desired set for all times when the distribution of $w_{t}$ is not fully known but lies within an ambiguity set, $\mathbb{D}_{t}$. The proposed distributionally robust safety specification tool will be used to design a controller for safety-critical stochastic systems under imperfect information about disturbance distributions.

To formulate a concrete safety specification problem, we consider a desired set $A$ for safety, which is an arbitrary compact Borel set in the state space $\mathbb{R}^{n}$. We also introduce the following definition of a probabilistic safe set:

Definition 1** (Probabilistic Safe Set).**

We define the probability that the system (2.1) is safe for all $t\in\bar{\mathcal{T}}$ given the strategy pair $(\pi,\gamma)$ and the initial value $\bm{x}$ as

[TABLE]

which we call the probability of safety for the set $A$. We also define the probabilistic safe set with probability $\alpha$ under $(\pi,\gamma)$ as the set

[TABLE]

This set contains all the initial states such that the probability that the system stays in the set $A$ is greater than or equal to $\alpha$ given the strategy pair $(\pi,\gamma)$. This definition generalizes the probabilistic safe set introduced in Abate et al. [1] to the case with ambiguous disturbance distributions. Using these notions, we now define a distributionally robust safe policy and set as follows:

Definition 2** (Distributionally Robust Safe Set).**

A control strategy $\pi^{\star}\in\Pi$ is said to be a distributionally robust safe policy given $x_{0}=\bm{x}$ if it satisfies

[TABLE]

The set $S_{\alpha}^{\star}(A)$ is said to be the distributionally robust safe set for $A$ with probability $\alpha$ if

[TABLE]

In other words, the distributionally robust safe policy $\pi^{\star}$ maximizes the worst-case probability of safety under distributional ambiguity characterized by the constraints in the set $\mathbb{D}_{t}$. No matter what form the strategy $\gamma$ takes so that the realized distribution lies in the ambiguity set $\mathbb{D}_{t}$, the probability that the system starting from $\bm{x}\in S_{\alpha}^{*}(A)$ stays safe is greater than or equal to $\alpha$ under the distributionally robust safe policy $\pi^{\star}$. Once we obtain $\pi^{\star}$ and $P_{\bm{x}}^{\mbox{\tiny safe}}$ for each $\bm{x}$, we can calculate the distributionally robust safe set $S_{\alpha}^{\star}(A)$ through simple thresholding. To this end, in the next subsection, we consider a game theoretic formulation to construct a distributionally robust safe policy.

2.3 A Dynamic Game Formulation

Let $\mathbb{1}_{A}:\mathbb{R}^{n}\to\{0,1\}$ be the indicator function of the set $A\subseteq\mathbb{R}^{n}$ such that $\mathbb{1}_{A}(\bm{x})=1$ if $\bm{x}\in A$ and $\mathbb{1}_{A}(\bm{x})=0$ otherwise. Then, given $x_{0}=\bm{x}$, we have

[TABLE]

where $\mathbb{E}^{\pi,\gamma}$ is the expectation taken with respect to the probability measure $\mathbb{P}^{\pi,\gamma}$ induced by the strategy pair $(\pi,\gamma)$. The problem of constructing a distributionally robust safe policy can then be formulated as the following zero-sum dynamic game problem:

[TABLE]

In this two-player game, Player I determines a control policy $\pi$ to maximize the probability of safety assuming that Player II selects a disturbance distribution strategy $\gamma$ to minimize the probability of safety. Recall that information about the ambiguity set $\mathbb{D}_{t}$ of probability distributions is encoded in Player II’s strategy space $\Gamma$. In general, the action space of Player II at each stage is infinite-dimensional because the disturbance may have a continuous probability density. Therefore, the Bellman equation associated with this dynamic game problem involves infinite-dimensional optimization problems, which are computationally challenging. To alleviate this computational issue, we propose a duality-based approach to reformulate the Bellman equation as a semi-infinite program that can be solved by convergent algorithms. In the next section, we first establish some analytical results about the dynamic game problem. In particular, we show that under mild conditions an associated value function is upper semi-continuous and a non-randomized Markov control policy is optimal.

3 Dynamic Programming

3.1 A Semi-Continuous Model

We let $\mathbb{K}_{t}\in\mathcal{B}(\mathbb{R}^{n}\times\mathbb{R}^{m}\times M_{+}(\mathbb{R}^{l}))$ be the collection of elements such that each $(\bm{x},\bm{u},\bm{\mu})\in\mathbb{K}_{t}$ satisfies $(i)$ $\bm{u}\in\mathbb{U}(\bm{x})$ and $(ii)$ $\bm{\mu}\in\mathbb{D}_{t}$. The stochastic kernel $Q_{t}(\bm{\xi}|\bm{x},\bm{u},\bm{\mu})$ represents the probability that the state of the system (2.1) at stage $t+1$ is equal to $\bm{\xi}\in\mathbb{R}^{n}$ given $(x_{t},u_{t},\mu_{t})=(\bm{x},\bm{u},\bm{\mu})$.

Assumption 1**.**

The problem (2.4) of constructing a distributionally robust safe policy satisfies the following conditions:

1. $(i)$

For each bounded continuous function $g_{t}:\mathbb{R}^{n}\to\mathbb{R}$, the function

[TABLE]

is continuous on $\mathbb{K}_{t}$ for each $t\in\mathcal{T}$. 2. $(ii)$

The set $\mathbb{U}(\bm{x})$ is compact for each $\bm{x}\in\mathbb{R}^{n}$. Furthermore, the set-valued mapping $\bm{x}\mapsto\mathbb{U}(\bm{x})$ is upper semi-continuous. 3. $(iii)$

The ambiguity set $\mathbb{D}_{t}$ is $\sigma$-compact for each $t\in\mathcal{T}$.

These conditions are standard measurable selection conditions for semi-continuous stochastic control models (e.g., [11], [19], [18]) and will be used to ensure the existence of a distributionally robust safe policy. We will further show that a non-randomized Markov policy is optimal. For each measurable function $\mathbb{v}$ on $\mathbb{R}^{n}$, we define a dynamic programming operator, denoted by $\mathbb{T}_{t}$, $t\in\mathcal{T}$, as

[TABLE]

We can first show the following properties of the dynamic programming operator. In particular, the outer “$\sup$” problem has an optimal solution under a mild condition on $\mathbb{v}$.

Lemma 1**.**

Let $\mathbb{v}:\mathbb{R}^{n}\to\mathbb{R}$ be a measurable upper semi-continuous function with $\|\mathbb{v}\|_{\infty}<\infty$ and $\mathbb{v}\geq 0$. Then,

1. $(i)$

$\mathbb{T}_{t}\mathbb{v}$* is upper semi-continuous.* 2. $(ii)$

There exists a measurable function $\kappa:\mathbb{R}^{n}\to\mathbb{R}^{m}$ such that, for all $\bm{x}\in\mathbb{R}^{n}$, $\kappa(\bm{x})\in\mathbb{U}(\bm{x})$ and

[TABLE]

Proof.

Define a function $\hat{\mathbb{v}}:\mathbb{R}^{n}\times\mathbb{R}^{m}\times M_{+}(\mathbb{R}^{l})\to\mathbb{R}$ as

[TABLE]

Since $\mathbb{v}$ is a measurable, upper semi-continuous and nonnegative function with finite $L^{\infty}$-norm, there exists a sequence $\{\mathbb{v}_{k}\}$ such that $\mathbb{v}_{k}\downarrow\mathbb{v}$ pointwise and each $\mathbb{v}_{k}$ is a bounded continuous function. Thus, for all $k$, $\int_{\mathbb{R}^{n}}\mathbb{v}(\xi)Q_{t}(d\xi|\bm{x},\bm{u},\bm{\mu})\leq\int_{\mathbb{R}^{n}}\mathbb{v}_{k}(\xi)Q_{t}(d\xi|\bm{x},\bm{u},\bm{\mu})$, and for any $(\bm{x}_{j},\bm{u}_{j},\bm{\mu}_{j})\to(\bm{x},\bm{u},\bm{\mu})$ we have that for all $k$

[TABLE]

due to Assumption 1 $(ii)$. Letting $k\to\infty$, we conclude that $\hat{\mathbb{v}}$ is upper semi-continuous. Since $\mathbb{1}_{A}:\mathbb{R}^{n}\to\mathbb{R}$ is upper semi-continuous with a compact set $A$, $(\bm{x},\bm{u},\bm{\mu})\mapsto\mathbb{1}_{A}(\bm{x})\hat{\mathbb{v}}(\bm{x},\bm{u},\bm{\mu})$ is upper semi-continuous as well. Furthermore, for all $(\bm{x},\bm{u},\bm{\mu})\in\mathbb{K}_{t}$,

[TABLE]

Thus, by Lemma 3.2. (b) in [18] have that $\mathbb{T}_{t}\mathbb{v}$ is upper semi-continuous and that there exists a measurable function $h:\mathbb{R}^{n}\to\mathbb{R}^{m}$ such that $\mathbb{T}_{t}\mathbb{v}(\bm{x})=\inf_{\bm{\mu}\in\mathbb{D}_{t}}\left[\mathbb{1}_{A}(\bm{x})\int_{\mathbb{R}^{n}}\mathbb{v}(\xi)Q_{t}(d\xi|\bm{x},\kappa(\bm{x}),\bm{\mu})\right]$ and $\kappa(\bm{x})\in\mathbb{U}(\bm{x})$ for all $\bm{x}\in\mathbb{R}^{n}$ under Assumption 1 $(iii)$ and $(iv)$. ∎

For each $t\in\mathcal{T}$, we define the value function of the distributionally robust safe control problem (2.4) as

[TABLE]

It represents the maximal worst-case probability of the system being safe from stage $t$ to $T$ when $x_{t}=\bm{x}$. For $t=T$, the value function is defined as $v_{T}(\bm{x})=\mathbb{1}_{A}(\bm{x})$. By definition, $v_{0}(x_{0})=\sup_{\pi\in\Pi}\inf_{\gamma\in\Gamma}P_{x_{0}}^{\mbox{\tiny safe}}(\pi,\gamma;A)$ for some initial state $x_{0}$. Setting $\mathbb{v}=v_{t+1}$ in Lemma 1, we can show that the distributionally robust safe control problem (2.4) admits a non-randomized Markov policy, which is optimal.

Theorem 1** (A Markov policy is optimal).**

Suppose that Assumption 1 holds. For each $t\in\mathcal{T}$, there exists a measurable function $\phi_{t}:\mathbb{R}^{n}\to\mathbb{R}^{m}$ such that $\phi_{t}(\bm{x})\in\mathbb{U}(\bm{x})$ and

[TABLE]

for all $\bm{x}\in\mathbb{R}^{n}$. The non-randomized Markov policy $\pi^{\star}:=(\phi_{0},\cdots,\phi_{T-1})\in\Pi$ is a distributionally robust safe policy, i.e.,

[TABLE]

Furthermore, the value function $v_{t}$ is upper semi-continuous for each $t\in\bar{\mathcal{T}}$.

Proof.

We first show that $v_{t}$ is a measurable upper semi-continuous function with $\|v_{t}\|_{\infty}<\infty$ for each $t$ by mathematical induction. For $t=T$, $v_{T}=\mathbb{1}_{A}$, which is upper semi-continuous because $A$ is closed. Furthermore, it is measurable and has a finite $L^{\infty}$-norm. Suppose now that $v_{t+1}$ is a measurable upper semi-continuous function with $\|v_{t+1}\|_{\infty}<\infty$ Then, by Lemma 1 $(i)$, $v_{t}=\mathbb{T}_{t}v_{t+1}$ is upper semi-continuous. Furthermore, it is clear that $v_{t}$ is measurable and $\|v_{t}\|_{\infty}\leq\|v_{t+1}\|_{\infty}<\infty$. This completes our inductive argument.

We now use Lemma 1 $(ii)$ to conclude that there exists a measurable function $\phi_{t}:\mathbb{R}^{n}\to\mathbb{R}^{m}$ such that $v_{t}(\bm{x})=\sup_{\bm{\mu}\in\mathbb{D}_{t}}[\mathbb{1}_{A}(\bm{x})\int_{\mathbb{R}^{n}}v_{t+1}(\xi)Q_{t}(d\xi|\bm{x},\phi_{t}(\bm{x}),\bm{\mu})]$ and $\phi_{t}(\bm{x})\in\mathbb{U}(\bm{x})$ for all $(t,\bm{x})\in\mathcal{T}\times\mathbb{R}^{n}$. Lastly, by the dynamic programming principle, we obtain that $v_{0}(\bm{x})=\inf_{\gamma\in\Gamma}\mathbb{E}^{\pi^{\star},\gamma}[\prod_{t=0}^{T}\mathbb{1}_{A}(x_{t})]$ with $x_{0}=\bm{x}$. ∎

This theorem greatly reduces the control strategy space we need to search for because it suffices to restrict our attention to non-randomized Markov policies. The Markov policy $\pi^{\star}$ maximizes the worst-case probability of safety no matter how the disturbance (Player II) chooses its probability distribution $\mu_{t}$ in the ambiguity set $\mathbb{D}_{t}$ using the history of information. Furthermore, the dynamic programming principle allows us to obtain the following Bellman equation:

Proposition 1**.**

Suppose that Assumption 1 holds. The value function $v_{t}$ solves the following Bellman equation: for $t=T$

[TABLE]

and for $t\in\mathcal{T}$

[TABLE]

Note that “$\max$” is used instead of “$\sup$” in the outer problem because its optimal solution exists due to Theorem 1. This Bellman equation is computationally challenging because it involves infinite-dimensional minimax optimization problems over the ambiguity set $\mathbb{D}_{t}$ of probability distributions. In Section 4, we propose a duality-based approach to alleviate the computational issue that arises from the infinite-dimensional minimax programs. Before introducing this method, we discuss how to construct distributionally robust safe sets and controllers through a stochastic reachability analysis in the following subsection.

3.2 Constructing Distributionally Robust Safe Sets and Safety-Oriented Controllers

To obtain distributionally robust control policies and safe sets, we evaluate the value function $\{v_{t}\}_{t=0}^{T}$ by solving the Bellman equation backward in time, i.e., from $t=T$ to $t=0$. A distributionally robust safe policy can then be constructed as

[TABLE]

The distributionally robust safe set with probability $\alpha$ can be computed as

[TABLE]

because $v_{0}(\bm{x})=\max_{\pi\in\Pi}\inf_{\gamma\in\Gamma}P_{\bm{x}}^{\mbox{\tiny safe}}(\pi,\gamma;A)$.

Define the $t$-distributionally robust safe set with probability $\alpha$ as

[TABLE]

If the system state lies in $S_{\alpha,t}^{\star}(A)$ at stage $t$, there exists a control policy such that the probability of the system being safe during the remaining stages is greater than equal to $\alpha$. Note that $S_{\alpha,0}^{\star}(A)=S_{\alpha}^{\star}(A)$ under Assumption 1 which guarantees the existence of a distributionally robust safe policy. Using the value function, we can compute the set as

[TABLE]

because $v_{t}(\bm{x})=\max_{\pi\in\Pi}\inf_{\gamma\in\Gamma}\mathbb{E}^{\pi,\gamma}[\prod_{s=t}^{T}\mathbb{1}_{A}(x_{s})]$ with $x_{t}=\bm{x}$. Suppose now that we are given the support $W_{t}$ of $\mu_{t}$ for each $t$. Consider the following controller: given $x_{t}$ at stage $t$, the control action is determined as

[TABLE]

This controller chooses an arbitrary admissible control action if it can drive the system into $S^{\star}_{\alpha,t+1}(A)$ at stage $t+1$ for any realization of the disturbance. Otherwise, it uses a distributionally robust safe policy. This procedure is motivated by the deterministic safe controller synthesis method of Lygeros et al. [28]. We can show that at each $t$ this controller ensures that the system will remain safe for stages $t+1,\cdots,T$ with probability greater than or equal to $\alpha$ under any disturbance distribution strategy $\gamma\in\Gamma$.

Proposition 2**.**

Suppose that Assumption 1 holds. If the initial state is chosen so that $x_{0}\in S_{\alpha}^{\star}(A)$ and the Markov control policy (3.2) is used, then

[TABLE]

i.e., the probability for the system being safe for all remaining stages is greater than or equal to $\alpha$, regardless of how the disturbance distribution is chosen in the ambiguity set.

Proof.

We use mathematical induction. For stage $t=0$, the statement is true since $x_{0}$ is assumed to be contained in $S_{\alpha}^{\star}(A)=S_{\alpha,0}^{\star}(A)$. Suppose that $x_{t}\in S_{\alpha,t}^{\star}(A)$ for some $t\in\mathcal{T}$. Fix an arbitrary $t\in\mathcal{T}$. Assume first that $x_{t}\in\{\bm{x}\>|\>f(\bm{x},\bm{u},\bm{w})\in S_{\alpha,t+1}^{\star}(A)\;\forall\bm{u}\in\mathbb{U}(\bm{x})\;\forall\bm{w}\in W_{t}\}$. Fix an arbitrary $u_{t}\in\mathbb{U}(x_{t})$. The controller guarantees that $x_{t+1}=f(x_{t},u_{t},w_{t})\in S_{\alpha,t+1}^{\star}(A)$ with probability 1 for any $\mu_{t}\in\mathbb{D}_{t}$ because ${\mu}_{t}(W_{t})=\int_{W_{t}}d{\mu}_{t}(\bm{w})=1$. On the other hand, when $x_{t}\notin\{\bm{x}\>|\>f(\bm{x},\bm{u},\bm{w})\in S_{\alpha,t+1}^{\star}(A)\;\forall\bm{u}\in\mathbb{U}(\bm{x})\;\forall\bm{w}\in W_{t}\}$, by using a distributionally robust safe policy $\phi_{t}^{\mbox{\tiny safe}}$, we can ensure that $x_{t+1}\in S_{\alpha,t+1}^{\star}(A)$ since

[TABLE]

where $\pi^{\mbox{\tiny safe}}:=\{\phi_{0}^{\tiny\mbox{safe}},\cdots,\phi_{T-1}^{\tiny\mbox{safe}}\}$ and $\mathcal{T}_{t}:=\{t,t+1,\cdots,T\}$. This completes our inductive argument. ∎

If another objective function needs to be minimized in a distributionally robust way while ensuring safety, one may solve

[TABLE]

to obtain an optimal distributionally robust policy $\pi^{opt}$, where $r:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}$ is a running cost function and $q:\mathbb{R}^{n}\to\mathbb{R}$ is a terminal cost function of interest. Then, one can employ the proposed controller (3.2) that chooses $\pi_{t}^{opt}(x_{t})$ whenever $x_{t}\in\{\bm{x}\>|\>f(\bm{x},\bm{u},\bm{w})\in S_{\alpha,t+1}^{\star}(A)\;\forall\bm{u}\in\mathbb{U}(\bm{x})\;\forall\bm{w}\in W_{t}\}$. Note that this controller prioritizes safety and tries to minimize the worst-case cost value whenever there is the flexibility to do so. This safety-oriented distributionally robust control design approach can be overly conservative, particularly when $W_{t}$ is large. However, it is computationally efficient because the cost-minimizing control problem is decoupled from the safe control problem.

4 Moment Uncertainty and Dual Bellman Equations

4.1 Ambiguity Sets with Moment Uncertainty

Recall that the proposed distributionally robust safe policies maximize the worst-case probability for a system to be safe, assuming that the probability distribution of the disturbance lies within an ambiguity set, $\mathbb{D}_{t}$, of distributions. Therefore, modeling the ambiguity set may critically affect the resulting safe policies. Several ambiguity set modeling approaches have been developed in the context of single-stage optimization problems. The approaches can be categorized as moment-based and statistical distance-based methods. A moment-based approach employs an ambiguity set of distributions whose moments (e.g., mean and covariance) satisfy certain constraints [43], [9], [35], [55], [50]. A statistical distance-based approach takes into account an ambiguity set of probability distributions that are closed to a nominal distribution in terms of a chosen statistical distance, such as $\phi$-divergence [3], [4], [22], [46], Prokhorov metric [14] and Wasserstein distance [33], [54], [15].

In this work, we take a moment-based approach. Suppose that an estimate of the mean and covariance matrix of the disturbance $w_{t}$ is the only available information. Let $\mathbb{m}_{t}\in\mathbb{R}^{l}$ and $\mathbb{\Sigma}_{t}\in\mathbb{R}^{l\times l}$ be the estimate of the mean and covariance matrix, respectively. The set of all the probability distributions (i.e., distribution measures) that are consistent with these estimates can be modeled as

[TABLE]

where $\mathbb{E}_{\mu_{t}}$ denotes the expectation taken with respect to the probability distribution $\mu_{t}$. Here, $b_{t}\in\mathbb{R}_{+}^{l}$ and $c_{t}\geq 1$ are given constants that depend on one’s confidence in the estimates $\mathbb{m}_{t}$ and $\mathbb{\Sigma}_{t}$. Any probability distribution in this set satisfies the following properties: $(i)$ the support of $w_{t}$ is $W_{t}$; $(ii)$ the mean of $w_{t,i}$ lies in a circle of size $b_{t,i}$; and $(iii)$ the centered second moment matrix of $w_{t}$ lies in a positive semidefinite cone. It models how likely $w_{t}$ is to be close to $\mathbb{m}_{t}$ in terms of the correlation matrix $c_{t}\mathbb{\Sigma}_{t}$ with $c_{t}\geq 1$.

4.2 Zero Duality Gap

In general, solving the Bellman equation (3.1) to evaluate $v_{t}(\bm{x})$ with the ambiguity set (4.1) is challenging as it involves infinite-dimensional minimax optimization problems. To resolve this difficulty, we propose a dual formulation method.111The proposed method does not resolve the scalability issue inherent in dynamic programming; the complexity of computing $v_{t}(\bm{x})$ is still exponential with the dimension of system state $\bm{x}$ even if the proposed approach is employed. Fix an arbitrary $(t,\bm{x},\bm{u})\in\mathcal{T}\times\mathbb{R}^{n}\times\mathbb{R}^{m}$ such that $\bm{u}\in\mathbb{U}(\bm{x})$. With the ambiguity set (4.1), the inner minimization problem in the Bellman equation (3.1) can be written as the following infinite-dimensional conic linear program:

[TABLE]

Here, (4.2b) and (4.2c) represent the first- and second-moment constraints encoded in the ambiguity set $\mathbb{D}_{t}$, respectively. The constraints (4.2d) and (4.2e) ensure that $\bm{\mu}$ is a probability distribution measure whose support is $\mathbb{D}_{t}$. Let $\underline{\bm{\lambda}},\overline{\bm{\lambda}}\in\mathbb{R}^{l}$ be the Lagrange multipliers associated with the inequality constraints (4.2b). We also let $\bm{\Lambda}\in\mathbb{S}^{l}$ and $\bm{\nu}\in\mathbb{R}$ be the Lagrange multipliers associated with (4.2c) and (4.2d), respectively. Its dual problem can then be derived as

[TABLE]

where $\underline{b}_{t}:=b_{t}-\mathbb{m}_{t}$ and $\overline{b}_{t}:=b_{t}+\mathbb{m}_{t}$. This is a semi-infinite program because the first constraint must be satisfied for all $\bm{w}\in W_{t}\subseteq\mathbb{R}^{l}$. Let $\inf\mathbf{P}$ and $\sup\mathbf{P}^{*}$ denote the optimal values of the primal and dual problems, respectively. By weak duality, we have $\sup\mathbf{P}^{*}\leq\inf\mathbf{P}$. However, we can further show that strong duality holds, i.e., the dual program is exact in the sense that the duality gap is zero.

Proposition 3** (Zero duality gap).**

Suppose that Assumption 1 holds, $\mathbf{P}$ has a feasible solution, and $W_{t}$ is compact. Then, $\mathbf{P}$ has an optimal solution and there is no duality gap, i.e.,

[TABLE]

Proof.

Note that $\mathbf{P}$ has a feasible solution with finite value since the objective function value lies in $[0,1]$. We introduce the following convex cone:

[TABLE]

Fix an arbitrary $\epsilon>0$. Choose $(\underline{\bm{\lambda}}^{\mbox{\tiny feas}},\overline{\bm{\lambda}}^{\mbox{\tiny feas}},\Lambda^{\mbox{\tiny feas}},\bm{\nu}^{\mbox{\tiny feas}},1)$ $\in P(W_{t})$ such that $\underline{\bm{\lambda}}^{\mbox{\tiny feas}}=\overline{\bm{\lambda}}^{\mbox{\tiny feas}}=0$, $\bm{\Lambda}^{\mbox{\tiny feas}}=\epsilon I$ and

[TABLE]

where

[TABLE]

Note that $\rho\in\mathbb{R}$ because $W_{t}$ is compact, and that $\bm{\nu}^{\mbox{\tiny feas}}\in\mathbb{R}$ because the value of $v_{t+1}$ is in $[0,1]$. Therefore, any $(\underline{\bm{\lambda}},\overline{\bm{\lambda}},\Lambda,\bm{\nu},{\lambda}_{0})$ that is contained in the $\epsilon$-ball centered at $(\underline{\bm{\lambda}}^{\mbox{\tiny feas}},\overline{\bm{\lambda}}^{\mbox{\tiny feas}},\Lambda^{\mbox{\tiny feas}},\bm{\nu}^{\mbox{\tiny feas}},1)$ lies in the cone $P(W_{t})$. This implies that $(\underline{\bm{\lambda}}^{\mbox{\tiny feas}},\overline{\bm{\lambda}}^{\mbox{\tiny feas}},\Lambda^{\mbox{\tiny feas}},\bm{\nu}^{\mbox{\tiny feas}},1)$ is an interior point of $P(W_{t})$. Thus, due to Theorem 1.2 in [24], there is no duality gap and the inf in $\mathbf{P}$ is attained.

∎

This proof ensures that a Slater type condition holds, and the rest follows from results of conic duality in infinite-dimensional convex optimization (see also [44]).

4.3 Dual Bellman Equation as a Semi-Infinite Program

Using the zero duality gap result, we can substitute the inner minimization problem ($\inf\mathbf{P}$) in the Bellman equation as its dual problem ($\sup\mathbf{P}^{*}$) without sacrificing optimality. This substitution allows us to evaluate the value function $v_{t}(\bm{x})$ via a dual version of the Bellman equation.

Theorem 2** (Dual Bellman equation).**

Suppose that Assumption 1 holds, $\mathbf{P}$ has a feasible solution, and $W_{t}$ is compact. For all $(t,\bm{x})\in\mathcal{T}\times\mathbb{R}^{n}$, the Bellman equation (3.1) is equivalent to the following semi-infinite program:

[TABLE]

with the terminal condition $v_{T}(\bm{x})=\mathbb{1}_{A}(\bm{x})$.

Remark 1**.**

For a compact representation, we merged “$\max_{\bm{u}}$” and “$\sup_{\underline{\bm{\lambda}},\overline{\bm{\lambda}},\bm{\Lambda},\bm{\nu}}$”. However, it should be noted that this semi-infinite program has an optimal feasible $\bm{u}$.

Since the dual problem $(\sup\mathbf{P}^{*})$ is a semi-infinite program, the dual Bellman equation also involves semi-infinite optimization problems. Each semi-infinite optimization program can be solved by using several convergent methods, such as discretization methods, exchange methods, homotopy methods, and primal-dual methods (e.g., [20], [42], [27] and the reference therein). In Section 5, we employ the discretization method proposed by Reemtsen [41]. This algorithm adaptively generates a grid on $W_{t}$ and converges to a locally optimal value of the semi-infinite program in the dual Bellman equation (see [41] for a proof). When the semi-infinite program is concave, it converges to the globally optimal value. We can show that each semi-infinite program is concave under the next assumption.

Assumption 2**.**

The following properties hold:

1. $(i)$

$f:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}^{l}\to\mathbb{R}^{n}$* is an affine function;* 2. $(ii)$

$A$* is a convex set;* 3. $(iii)$

For all $\bm{x}_{1},\bm{x}_{2}\in\mathbb{R}^{n}$ and for all $\lambda\in(0,1)$, if $\bm{u}_{i}\in\mathbb{U}(\bm{x}_{i})$, $i=1,2$, then $\lambda\bm{u}_{1}+(1-\lambda)\bm{u}_{2}\in\mathbb{U}(\lambda\bm{x}_{1}+(1-\lambda)\bm{x}_{2})$.

When $\mathbb{U}$ is independent of $\bm{x}$, Assumption 2 $(iii)$ is equivalent to the concavity of $\mathbb{U}$. Note that this assumption is independent of the ambiguity set $\mathbb{D}_{t}$. Thus, the following concavity result holds with general ambiguity sets:

Proposition 4**.**

Suppose that Assumptions 1 and 2 hold. Then, the value function $v_{t}(\bm{x})$ is concave with respect to $\bm{x}\in A$ for each $t\in\mathcal{T}$.

Its proof is contained in the Appendix. Since $\bm{u}\mapsto v_{t}(f(\bm{x},\bm{u},\bm{w}))$ is concave for each $(t,\bm{x},\bm{w})\in\bar{\mathcal{T}}\times A\times W_{t}$ under Assumption 2, the semi-infinite program in the dual Bellman equation is concave for each $(t,\bm{x})\in\mathcal{T}\times A$.

Remark 2**.**

When control actions are chosen from a discrete set, i.e., $\mathbb{U}(\bm{x})$ is a discrete set, the semi-infinite program in Theorem 2 can be formulated as a mixed-integer program, where “$\max_{\bm{u}}$” is a discrete-optimization problem and “$\sup_{\underline{\bm{\lambda}},\overline{\bm{\lambda}},\bm{\Lambda},\bm{\nu}}$” is a continuous optimization problem. In particular, under Assumption 2, we can employ a linear approximation-based method to obtain an approximate solution with a provable suboptimality bound [52].

5 Numerical Examples: Thermostatically Controlled Loads

Thermostatically controlled loads (TCLs)—such as air conditioners, refrigerators, water heaters, and battery pack cooling systems—are used to guarantee human comfort, and food provision and battery safety, etc. Therefore, ensuring safe TCL operation is critical in a wide range of applications. We consider the following model of the temperature being controlled through a TCL:

[TABLE]

which was originally developed by Mortensen and Haggerty [34]. Here, $x_{t}\in\mathbb{R}$ is the temperature of interest (e.g., indoor temperature, food temperature), $u_{t}\in\{0,1\}$ is an ON/OFF control input, and $w_{t}\in\mathbb{R}$ is a disturbance variable that takes into account environmental and/or human behavioral uncertainty. Note also that $\alpha=\exp(-h/CR)$, where $C$ is the thermal capacitance (kWh/∘C), $R$ is the thermal resistance (∘C/kW) and $h$ is the time interval between stages $t$ and $t+1$. In addition, the parameter $P$ represents the range of energy transfer to or from the thermal mass (kW) and the coefficient $\eta$ is the control efficiency. In our numerical experiments, the following parameters are used: $R=2$ ∘C/kW, $C=2$ kWh/∘C, $\theta=32$ ∘C, $h=5/60$ hour, $P=14$ kW, and $\eta=0.7$. We compute the distributionally robust safe sets and policies for $90$ minutes with 5-minute time intervals between two consecutive stages. The desired safe set of temperature values is chosen as $A=[19,22]$ (∘C).

We now assume that an empirical estimate of the first- and second-moments is the only available information about $w_{t}$ except its support. Let $\mathbb{m}$ and $\mathbb{\Sigma}$ be the empirical mean and variance of $w_{t}$. As discussed in Section 4.1, it is reasonable to consider the following constraints: $|\mathbb{E}_{\mu_{t}}[w_{t}]-\mathbb{m}|\leq b$, and $\mathbb{E}_{\mu_{t}}[(w_{t}-\mathbb{m})^{2}]\leq c\mathbb{\Sigma}$, where $b\geq 0$ and $c\geq 1$ are adjustable parameters depending on one’s confidence in the estimate. We call them confidence parameters. We set $\mathbb{m}=0$, $\mathbb{\Sigma}=0.25^{2}$, and the disturbance’s support $\mathcal{K}=[-\frac{1}{2}\sqrt{\mathbb{\Sigma}/12},\frac{1}{2}\sqrt{\mathbb{\Sigma}/12}]$.

5.1 Effect of the Confidence Parameters

Fig. 2 shows the probability $P_{\bm{x}}^{\mbox{\tiny safe}}$ of safety as a function of the initial state $\bm{x}\in[18,23]$ for multiple confidence parameters $b$ and $c$. The function has a bimodal structure due to the discrete ON/OFF control inputs: it can be considered as the point-wise maximum of the probability function with OFF control and that with ON control. At initial states between the two peaks, ON or OFF control input at time [math] does not maximize the probability of safety. Aditionally, as the initial state approaches the boundary of the set $A=[19,22]$, the probability of safety decreases.

As shown in Fig. 2 (a), given $c=1$, the probability of safety decreases with the confidence parameter $b$ for the mean. In other words, an inaccurate mean estimate $\mathbb{m}$ makes it difficult for the system to remain safe for all stages. Fig. 2 (b) illustrates that the probability of safety decreases as the uncertainty of the variance estimate $\mathbb{\Sigma}$ increases when $b=0.1$. The probability $P_{\bm{x}}^{\mbox{\tiny safe}}$ of safety scales down with the variance confidence parameter $c$ without any change in its support. On the other hand, an increase in the mean confidence parameter $b$ reduces the support of $P_{\bm{x}}^{\mbox{\tiny safe}}$. This is because a change in $b$ may shift the worst-case disturbance distribution while a change in $c$ can only scale the worst-case distribution.

5.2 Safety-Oriented Distributionally Robust Control

To demonstrate the performance of the proposed safety-oriented distributionally robust controller, we compare it to a safety-oriented controller synthesized with standard probabilistic safe sets. We use the safety-oriented controller design approach proposed in Section 3.2. When the control action is allowed to be arbitrarily chosen in (3.2), we choose OFF control input to minimize the energy cost. In other words, it is an energy cost-minimizing safety-oriented controller. Suppose that the true disturbance distribution is uniformly distributed over the support $\mathcal{K}$ with mean $\mathbb{m}$, and variance $\mathbb{\Sigma}$. We consider the situation in which we misestimate the distribution as a truncated normal distribution with the same support $\mathcal{K}$, mean $\mathbb{m}$ and variance $\mathbb{\Sigma}/2$.222The variance of the truncated normal distribution must be greater than the variance of the uniform distribution with the same support and mean. We set the probability threshold as $\alpha=0.95$ and construct the probabilistic safe sets using the method proposed by Abate et al. [1] with the inaccurately estimated distribution. As shown in Fig. 3 (a), the safety-oriented controller obtained using the probabilistic safe sets fails to guarantee that the probability of safety will be greater than or equal to the threshold $\alpha=0.95$. Specifically, in our numerical experiment with the initial state $\bm{x}=21$, $1$,$365$ of $10$,$000$ sample trajectories violated the safety constraints. Thus, the probability of the system being safe for all stages is only $0.8635$ even though the proposed safety-oriented controller is constructed in a conservative way for safety. However, when the distributionally robust safe sets are employed to construct the safety-oriented controllers, only $5$ sample trajectories move out from the set $A=[19,22]$. In other words, the probability of safety is $0.995$.333The resulting probability of safety is significantly greater than the threshold $0.95$ because the uniform distribution is not the worst-possible distribution.

6 Conclusions

We proposed a dynamic game approach to computing distributionally robust safe sets and policies concerning ambiguous information about the probability distribution of disturbances. We identified conditions under which a Markov policy is an optimal distributionally robust safe policy. Such a policy leads to a practical design method for safety-oriented stochastic controllers that guarantee that the probability of the system being safe for all remaining stages exceeds a pre-specified threshold, regardless of how the disturbance distribution is chosen in an ambiguity set. We also proved that there is no duality gap in the inner minimization problem of the Bellman equation when ambiguity sets with moment uncertainty are considered. This strong duality result allows us to reformulate the infinite-dimensional minimax problem in the Bellman equation as a semi-infinite program. Since the reformulated dual Bellman equation can be solved using existing convergent algorithms, the proposed dual formulation method alleviates computational issues that otherwise occur when solving the distributionally robust safety specification problem. Through numerical simulations, we demonstrated that the safety-oriented controller constructed using the distributionally robust safe sets and policies can guarantee the desired probability of safety even when the probability distribution of disturbances is inaccurately estimated; meanwhile, the same controller based on standard probabilistic safe sets cannot.

The following future directions are of interests to generalize and improve the proposed approach. First, this method is readily applicable to backward reachability analysis. Given a target set $B$, the probability that reaches $B$ for some $t\in\bar{\mathcal{T}}$ given the strategy pair $(\pi,\gamma)$ and the initial value $\bm{x}$ can be expressed as $P_{\bm{x}}^{\mbox{\tiny reach}}(\pi,\gamma;B)=\mathbb{E}^{\pi,\gamma}[\max_{t\in\bar{\mathcal{T}}}\mathbb{1}_{B}(x_{t})]$. We can then use dynamic programming and the same dual formulation to obtain a control policy that maximizes this probability. Similarly, the proposed distributionally robust approach can be extended to reach-avoid problems. Second, the proposed method is compatible with other types of ambiguity sets. The existence result and safety-oriented controller design approach remain valid when different ambiguity sets are considered. However, different reformulations of associated Bellman equations are required to handle other ambiguity sets such as statistical distance-based ones by using different existing strong duality results. Third, this method presents a scalability issue arising from both dynamic programming and semi-infinite programming. Applying and developing advanced computational techniques including approximate dynamic programming, and sampling- and moment-based approximation methods is of great interests to alleviate the scalability issue.

Appendix A Proof of Proposition 4

We use mathematical induction to prove this proposition. For $t=T$, $v_{T}(\bm{x})=\mathbb{1}_{A}(\bm{x})$ is concave with respect to $\bm{x}\in A$. Suppose that $\bm{x}\mapsto v_{s}(\bm{x})$ is concave for $\bm{x}\in A$ for $s=T-1,\cdots,t+1$. Fix arbitrary $\bm{x}^{1},\bm{x}^{2}\in\mathbb{R}^{n}$ and $\lambda\in(0,1)$. Let $\bm{u}^{i}\in\mathbb{U}(\bm{x}^{i})$ be a solution to the outer maximization problem in (3.1) at $(t,\bm{x}^{i})$ for $i=1,2$. We also let $\bm{x}^{\lambda}:=\lambda\bm{x}^{1}+(1-\lambda)\bm{x}^{2}$ and $\bm{u}^{\lambda}:=\lambda\bm{u}^{1}+(1-\lambda)\bm{u}^{2}$. Since $A$ is a convex set, $\bm{x}^{\lambda}\in A$. In addition, Assumption 2 $(iii)$ implies that $\bm{u}^{\lambda}\in\mathbb{U}(\bm{x}^{\lambda})$. Therefore, we have that

[TABLE]

Since $f$ is an affine function, for each $\bm{w}\in\mathbb{R}^{l}$, $f(\bm{x}^{\lambda},\bm{u}^{\lambda},\bm{w})=\lambda f(\bm{x}^{1},\bm{u}^{1},\bm{w})+(1-\lambda)f(\bm{x}^{2},\bm{u}^{2},\bm{w})$. Combining this with the concavity of $v_{t+1}$, we further have that

[TABLE]

which implies that $v_{t}$ is concave. This completes our inductive argument. ∎