Risk-averse risk-constrained optimal control

TL;DR

This paper introduces a decomposition approach for multistage risk-averse optimal control problems with nested risk mappings, enabling more efficient solutions and accounting for uncertainty propagation over time.

Contribution

It presents a novel decomposition method for complex nested risk-averse control problems and introduces a new form of risk constraints considering temporal uncertainty propagation.

Findings

Enables efficient numerical solutions for nested risk-averse control problems.

Introduces a new risk constraint form that accounts for uncertainty over time.

Improves robustness of control strategies against low-probability events.

Abstract

Multistage risk-averse optimal control problems with nested conditional risk mappings are gaining popularity in various application domains. Risk-averse formulations interpolate between the classical expectation-based stochastic and minimax optimal control. This way, risk-averse problems aim at hedging against extreme low-probability events without being overly conservative. At the same time, risk-based constraints may be employed either as surrogates for chance (probabilistic) constraints or as a robustification of expectation-based constraints. Such multistage problems, however, have been identified as particularly hard to solve. We propose a decomposition method for such nested problems that allows us to solve them via efficient numerical optimization methods. Alongside, we propose a new form of risk constraints which accounts for the propagation of uncertainty in time.

Figures

Equations

$${\text{E}}^{\pi }[Z]={\pi }^{\top }Z.$$

$$\max [Z]=\underset{i=1,\dots ,n}{\max }{Z}_i.$$

$${\rho }^{\lambda }[Z]=(1-\lambda )\rho [Z]+\lambda {\text{E}}^{\pi }[Z],$$

$$\rho [Z]=\underset{\mu \in \text{Af}(\pi )}{\max }{\text{E}}^{\mu }[Z],$$

$${\text{AVAR}}_{\alpha }[Z]=\{\begin{array}{ll}\underset{t\in \Re }{\min }t+\text{nicefrac}1\alpha {\text{E}}^{\pi }\text{plus}Z-t,& \alpha \ne 0\\ \max [Z],& \alpha =0\end{array}$$

$${\text{Af}}_{\alpha }^{\mathrm{avar}}(\pi )=\left\{\mu \in {\Re }^n|{\sum }_{i=1}^n{\mu }_i=1,0\le {\mu }_i\le \frac{{\pi }_i}{\alpha }\right\}.$$

$${\text{Af}}_{\alpha }^{\mathrm{evar}}(\pi )=\left\{\mu \in {\mathcal{D}}_n\mid D_{\mathrm{KL}}(\mu \Vert \pi )\le -\ln \alpha \right\},$$

$$D_{\mathrm{KL}}(\mu \Vert \pi )\text{dfn}\sum _{i=1}^n{\mu }_i\ln \frac{{\mu }_i}{{\pi }_i},$$

$$x_{t+1}=f(x_t,u_t,w_t),$$

$$x^{i_{+}}=f(x^i,u^i,w^{i_{+}})$$

$${\rho }_{\mid t}[Z_t]=({\rho }^i[Z^{[i]}]{)}_{i\in \mathrm{nodes}(t)}.$$

$$\rho_{\mid{}t}[Z_{t}]{}={}\Big{(}\max_{\mu^{i}\in\Af^{i}(\pi^{[i]})}\E^{\mu^{i}}[Z^{[i]}]\Big{)}_{i\in\operatorname{nodes}(t)},$$

$$\varrho_{t}(Z_{0},\dots,Z_{t}){}={}\rho_{\mid{}0}\big{[}Z_{0}{}+{}\rho_{\mid{}1}[{}\cdots{}{}+{}\rho_{\mid{}t}[{}Z_{t}{}]{}]\big{]}$$

$${\bar{\rho }}_t[Z_t]={\varrho }_k(0,\dots ,0,Z_t).$$

$$V^{\star }=$$

$$+$$

$$x_{t+1}=$$

$$r_t[{\varphi }_{j,t}(x_t,u_t,w_t)]\le$$

$$r_N[{\varphi }_{j,N}(x_N)]\le$$

$$r_t[G_{j,t}]\le 0,$$

$$\bar{r}_{t}[{}G_{j,t}{}]{}={}r_{\mid{}0}\big{[}r_{\mid{}1}\big{[}\cdots{}r_{\mid{}t}[{}G_{j,t}{}]\big{]}\big{]}{}\leq{}0$$

$$\rho [Z]=\underset{\mu \in {\Re }^n,\nu \in {\Re }^r}{\max }\{{\mu }^{\top }Z\mid E\mu +F\nu {\preccurlyeq }_{\mathcal{K}}b\},$$

$$\mathcal{A}=\{\mu \in {\Re }^n\mid \exists \nu \in {\Re }^r:E\mu +F\nu {\preccurlyeq }_{\mathcal{K}}b\}.$$

$$\displaystyle\Af^{\mathrm{evar}}_{\alpha}(\pi){}={}\left\{\mu\in\mathcal{D}_{n}\left|{\begin{array}[]{l}\exists\nu\in\Re^{n}:\sum_{i=1}^{n}\nu_{i}{}\leq{}-\ln\alpha,\\ (-\nu_{i},\mu_{i},\pi_{i}){}\in{}\mathcal{K}^{\mathrm{exp}},i\in\N_{[1,n]}\end{array}}\right.\right\}{\!,}$$

$$\mathcal{A}^{\lambda}{}={}\left\{\mu\in\mathcal{D}_{n}\left|{\begin{array}[]{l}\exists\nu\in\Re^{r+n}\text{ such that:}\\ \smallmat{0\\ I}\mu+\smallmat{E&F\\ (\lambda-1)I&0}\nu\preccurlyeq_{\mathcal{K}\times\{0_{n}\}}\smallmat{b\\ \lambda\pi}\end{array}}\right.\right\}.$$

$$\rho [Z]=\underset{y}{\min }\{y^{\top }b\mid E^{\top }y=Z,F^{\top }y=0,y{\succcurlyeq }_{{\mathcal{K}}^{\ast }}0\}.$$

$$\displaystyle\rho\Big{[}\inf_{x\in X}g(x)\Big{]}{}={}\inf_{x\in X}\rho[{}g(x){}]$$

$$\underset{x\in X}{\mathrm{argmin}}g(x)\subseteq \underset{x\in X}{\mathrm{argmin}}\rho [g(x)].$$

$$\underset{x\in X}{\mathrm{argmin}}g(x)=\underset{x\in X}{\mathrm{argmin}}\rho [g(x)].$$

$$\operatorname{epi}\rho=\left\{(Y,\gamma)\in\Re^{n{+}1}{}\left|{}\begin{array}[]{l}\exists y\succcurlyeq_{\mathcal{K}^{*}}0,E^{{\scriptscriptstyle\top}}y{=}Y,\\ F^{{\scriptscriptstyle\top}}y=0,y^{{\scriptscriptstyle\top}}b\leq\gamma\end{array}{}\right.{}\right\}.$$

$$\mathrm{epi}{\rho }_{\mid t}={\prod }_{i\in \mathrm{nodes}(t)}\mathrm{epi}{\rho }^i$$

$$\operatorname{epi}\bar{\rho}_{t}{\,}{=}{\!}\left\{\begin{array}[]{l}(Y_{t+1},Y_{0})\in\Re^{|\operatorname{nodes}(t+1)|+1}{}\mid{}\exists(Y_{j})_{j\in\N_{[1,t]}},\\ Y_{j}\in\Re^{\operatorname{nodes}(j)},(Y_{j+1},Y_{j}){\in}\operatorname{epi}\rho_{\mid{}j},\,j\in\N_{[0,t]}\end{array}\right\}$$

Full text

I Introduction

I-A Background, motivation and contributions

Risk measures in stochastic optimal control serve two purposes: firstly, they allow to account for inexact knowledge of the underlying probability distribution — which in most cases is merely estimated — and, secondly, offer a flexible framework which interpolates between worst-case and expectation-based (risk-neutral) formulations [1, 2]. Risk-averse optimal control aims at optimizing the expectation of a (random) cost function accounting for the worst-case probability distribution — a general approach which has been termed distributionally robust optimization [3].

In several applications it is desirable to impose constraints on random quantities in a probabilistic fashion (typically in the form of probabilistic or expectation constraints), yet in doing so one should take into account the ambiguity associated with the probability distribution [4]. Risk constraints can be interpreted as ambiguous expectation constraints [5, 6] and are often employed as surrogates for chance constraints [7, 8] in order to avoid having to resort to computationally demanding methods such as integer programming [9].

Risk-averse optimal control formulations are nowadays making their way in applications such as power systems [10], and economics [2] as their favorable properties are becoming evident. Yet, their applicability is hindered by their complexity and computational cost of associated multistage formulations. Multistage risk-averse optimal control problems amount to the optimization of a composition of several nonsmooth mappings [1]. Typical numerical solution approaches, such as stochastic dual dynamic programming, fall short when faced with large dimension of scenario-based problems [11, 12, 13]. When the involved risk measures are of the average value-at-risk type, we may obtain explicit solutions by multiparametric piecewise quadratic programming; this is, however, limited to systems with with few states and small prediction horizons [14].

The contributions of this paper are twofold: (i) we present a novel framework for risk constraints using nested risk measures which aim at accounting for the propagation of ambiguity; we call this new type of risk constraints, multistage nested risk constraints, (ii) we propose a reformulation of multistage risk-averse problems involving nested risk measures which facilitates their numerical solution.

While much of the research attention has focused on particular risk measures, such as the average value-at-risk [15, 8] and the mean upper semi-deviation [16], it has been unclear how to extend existing results to more general risk measures. Overall, our approach makes use of the dual conic representation of risk measures and allows for the use of arbitrary coherent risk measures in contrast to existing approaches which focus on specific risk measures.

I-B Notation

Let $\N_{[k_{1},k_{2}]}$ denote the integers in $[k_{1},k_{2}]$. For $z\in\Re^{n}$ let $\plus{z}=\max\{0,z\}$, where the max is taken element-wise. We denote the transpose of a matrix $A$ by $A^{{\scriptscriptstyle\top}}$. The dual cone $\mathcal{K}^{*}$ of a closed convex cone $\mathcal{K}\subseteq\Re^{n}$ is the set $\mathcal{K}^{*}=\{y\in\Re^{n}\mid y^{\top}x\geq 0,\forall x\in\mathcal{K}\}$. The relative interior of $\mathcal{K}$ is denoted by $\operatorname{ri}(\mathcal{K})$. A function $f:\Re^{n}\to\Re$ is called lower semicontinuous (lsc) if its lower level sets, $\{x{}\mid{}f(x)\leq\alpha\}$, are closed and it is called level bounded if its lower level sets are bounded.

II Measuring risk

Let $\Omega=\{\omega_{i}\}_{i=1}^{n}$ be a finite sample space equipped with the discrete $\sigma$-algebra $2^{\Omega}$ and a probability measure $\prob$ with $\prob(\{\omega_{i}\})=\pi_{i}$. Without loss of generality, let us assume that $\pi_{i}>0$. The pair $(\Omega,\prob)$ is called a probability space. A vector $p\in\Re^{n}$ is called a probability vector if $p_{i}\geq 0$ for all $i\in\N_{[1,n]}$ and $\sum_{i=1}^{n}p_{i}=1$. The set of all probability vectors in $\Re^{n}$ is called the probability simplex and is denoted by $\mathcal{D}_{n}$. A real-valued random variable over $(\Omega,\prob)$ is a mapping $Z:\Omega\to\Re$ with $Z(\omega_{i})=Z_{i}$; this can be identified by the vector $Z=(Z_{1},\ldots,Z_{n})\in\Re^{n}$.

Suppose that $Z$ corresponds to a random cost. One possible way to extract a characteristic index out of $Z$ which quantifies its magnitude is to compute its expectation, which is

[TABLE]

However, the expectation carries no deviation information and may fail to take into account extreme outcomes of the cost which might happen with low probability. At the opposite end, the maximum of $Z$ is defined as

[TABLE]

However, the maximum disregards the probability distribution and is likely to produce very conservative values.

Risk measures are mappings $\rho:\Re^{n}\to\Re$ which are used to derive a sure outcome which is no worse than $Z$ itself. In other words, a risk measure extracts a characteristic index taking into account the significance of high costs, which may happen with low probability. Trivially, the expectation and the maximum are risk measures.

A risk measure $\rho:\Re^{n}\to\Re$ is said to be convex if for all $Z,Z^{\prime}\in\Re^{n}$, $c\in\Re$, $\lambda\in[0,1]$ the following properties hold

1. A1.

Convexity. $\rho[\lambda Z+(1-\lambda)Z^{\prime}]\leq\lambda\rho[Z]+(1-\lambda)\rho[Z^{\prime}]$, 2. A2.

Monotonicity. $\rho[Z]\leq\rho[Z^{\prime}]$, whenever $Z_{i}\leq Z^{\prime}_{i}$ for all $i\in\N_{[1,n]}$, 3. A3.

Translation equivariance. $\rho[Z+c1_{n}]=\rho[Z]+c$.

A convex risk measure is called coherent if it satisfies the additional axiom [1, Def. 6.4]

1. A4.

Positive homogeneity. $\rho[\alpha Z]=\alpha\rho[Z]$ for all $\alpha\geq 0$.

Coherent risk measures are considered well behaving and the coherency axioms are heavily exploited in risk-averse optimization formulations. Certain risk measures may satisfy a stronger monotonicity assumption [17, 18, 1]

1. A5.

Strict/strong monotonicity. The risk measure $\rho$ is called strictly (strongly) monotone if $\rho[Z]<\rho[Z^{\prime}]$ whenever $Z\leq Z^{\prime}$, $\prob[Z<Z^{\prime}]>0$ (and $\max Z<\max Z^{\prime}$).

A monotone risk measure $\rho$ can be regularized to produce a strictly monotone risk measure by defining

[TABLE]

for $\lambda\in(0,1]$. Additionally, $\rho^{\lambda}$ preserves the coherency of $\rho$.

An important duality result is that all coherent risk measures can be written as [1, Thm. 6.5]

[TABLE]

where $\Af(\pi){}\subseteq{}\mathcal{D}_{n}$ is a closed and convex set of probability vectors which contains $\pi$. We call $\Af(\pi)$ the ambiguity set of $\rho$. Equation (1) offers an interpretation of coherent risk measures: a coherent risk measure is the worst-case expectation under inexact knowledge of the underlying probability vector $\mu$. For example, $\max[Z]=\max_{\mu\in\mathcal{D}_{n}}\E^{\mu}[Z]$, that is, the maximum operator reflects the total lack of probabilistic information.

A popular risk measure is the average value-at-risk with parameter $\alpha\in[0,1]$, which is defined as

[TABLE]

The ambiguity set of $\AVARa$ is

[TABLE]

The average value-at-risk is a coherent, non-strongly monotone (except for $\alpha=1$), risk measure. For $\alpha=1$, $\Af^{\mathrm{avar}}_{1}(\pi)=\{\pi\}$ and $\AVAR_{1}[Z](\pi)=\E[Z]$. The maximal ambiguity set is attained for $\alpha=0$, i.e., $\Af^{\mathrm{avar}}_{0}{}={}\mathcal{D}_{n}$. Therefore, $\AVARa$ interpolates between the risk-neutral expectation operator ($\AVAR_{1}$) and the worst-case maximum ($\AVAR_{0}$).

Risk measures with polytopic ambiguity sets can be fully described by the set of vertices of their ambiguity sets, that is $\Af=\operatorname{conv}\{\mu^{(l)}\}_{l=1}^{\kappa}$ and $\rho[Z]{}={}\max_{l\in\N_{[1,\kappa]}}\E^{\mu^{(l)}}[Z]$. However, the computation of these extreme points is a computationally demanding operation111Let $\Omega=\{\omega_{i}\}_{i=1}^{n}$ with $\pi_{i}=\nicefrac{{1}}{{n}}$. For $n=10$, the minimal representation of $\mathcal{A}(\pi)$ counts $252$ vertices. For $n=15$, the number of vertices increases to $51480$. For $n{}={}13$, the determination of the minumum number of vertices by the MPT toolbox (using Gurobi), requires $3.7$ hours..

Another popular coherent and strongly monotone risk measure is the entropic value-at-risk at level $\alpha\in(0,1]$, denoted by $\EVARa$ [19], whose ambiguity set is given by

[TABLE]

where

[TABLE]

is the Kullback-Leibler divergence from $\pi$ to $\mu$. We have $\EVAR_{1}[Z]=\E[Z]$ and $\lim_{\alpha\downarrow 0}\EVARa[Z]=\max[Z]$. The entropic value-at-risk $\EVARa$ is strongly monotone and, additionally, strictly monotone over the space of random variables with $\prob[\{\omega{}\mid{}Z(\omega)=\max[Z]\}]<1-\alpha$ [17].

III Stochastic systems and multistage risk

III-A System dynamics and scenario trees

Consider the following discrete-time dynamical system

[TABLE]

with state variable $x_{t}\in\Re^{n_{x}}$, input $u_{t}\in\Re^{n_{u}}$ and a disturbance $w_{t}\in\Re^{n_{w}}$ which is a random process. We will study the evolution of this system throughout a finite horizon of $N$ future time instants referred to as stages. Starting from a known initial state $x_{0}$ the system states evolve according to (5) as illustrated in Figure 2(a) giving rise to a structure known as a scenario tree. There have been proposed several methodologies to generate scenario trees from data [20, 21].

The nodes of the tree are assigned a unique index $i$ with $i=0$ being the root node which corresponds to the initial state $x_{0}$. The nodes at stage $t\in\N_{[0,N]}$ are denoted by $\operatorname{nodes}(t)$. Starting from the root node, a node $i$ is visited with probability $\pi^{i}>0$ (and $\pi^{0}=1$); this makes $\operatorname{nodes}(t)$ a probability space with probability vector $\pi_{t}=(\pi^{i})_{i\in\operatorname{nodes}(t)}$.

The unique ancestor of a node $i\in\operatorname{nodes}(t)\setminus\{0\}$ is denoted by $\operatorname{anc}(i)$ and the set of children of $i\in\operatorname{nodes}(t)$ for $t\in\N_{[0,N-1]}$ is $\operatorname{child}(i)\subseteq\operatorname{nodes}(t+1)$; this becomes a probability space with probability vector $\pi^{[i]}=\frac{1}{\pi^{i}}(\pi^{i_{+}})_{i_{+}\in\operatorname{child}(i)}$.

As shown in Fig. 2(a), every node $i$ of the tree is associated with a state value $x^{i}$ and all non-leaf nodes $i$ are assigned an input $u^{i}$. Every edge connecting $i$ with $i_{+}{}\in{}\operatorname{child}(i)$ is associated with a disturbance $w^{i_{+}}$. The finite-horizon evolution of (5) on the scenario tree is described by

[TABLE]

for all $i\in\operatorname{nodes}(t)$, $t\in\N_{[0,N-1]}$ and $i_{+}\in\operatorname{child}(i)$. Note that having assigned a control action $u^{i}$ means that decisions are made in a causal fashion, i.e., control actions $u_{t}$ are only allowed to depend on information that is available up to time $t$. The nodes of the tree at stage $N$ are called leaf nodes.

III-B Measuring risk on scenario trees

In this section we introduce the notion of conditional risk mappings which is essential in measuring the risk of a random cost which evolves in time across the nodes of a scenario tree [1, Sec. 6.8.1].

For $t\in\N_{[0,N-1]}$, let $\ell_{t}:\Re^{n_{x}}\times\Re^{n_{u}}\times\Re^{n_{w}}\to\Re$ be a stage cost function and $\ell_{N}:\Re^{n_{x}}\to\Re$ be the terminal cost function. Such cost functions will be used in the following section to construct optimal control problems.

Every node $i\in\operatorname{nodes}(t+1)$, $t\in\N_{[0,N-1]}$ is associated with a cost value $Z^{i}{}={}\ell_{t}(x^{\operatorname{anc}(i)},u^{\operatorname{anc}(i)},w^{i})$. For each $t\in\N_{[0,N-1]}$ we define a random variable $Z_{t}=(Z^{i})_{i\in\operatorname{nodes}(t+1)}$ on the probability space $\operatorname{nodes}(t+1)$. For example, the cost at $t=0$ is the random variable $Z_{0}=(Z^{i})_{i\in\operatorname{nodes}(1)}=(\ell_{0}(x^{0},u^{0},w^{i}))_{i\in\operatorname{nodes}(1)}$. At stage $N$ the terminal cost is the random variable $Z_{N}=(\ell_{N}(x^{i}))_{i\in\operatorname{nodes}(N)}$.

By defining $Z^{[i]}{}\dfn{}(Z^{i_{+}})_{i_{+}\in\operatorname{child}(i)}$, $i\in\operatorname{nodes}(t)$, we partition the variable $Z_{t}=(Z^{[i]})_{i\in\operatorname{nodes}(t)}$ into groups of nodes which share a common ancestor as shown in Fig. 2(b).

Let $\rho^{i}:\Re^{|\operatorname{child}(i)|}{\to}\Re$ be risk measures on the probability space $\operatorname{child}(i)$. For every stage $t\in\N_{[0,N-1]}$ we may define a conditional risk mapping at stage $t$, $\rho_{\mid{}t}:\Re^{|\operatorname{nodes}(t+1)|}\to\Re^{|\operatorname{nodes}(t)|}$, as follows

[TABLE]

This construction is illustrated in Fig. 2(b).

Conditional risk mappings admit a dual representation akin to that in Eq. (1). Provided that all $\rho^{i}$ are coherent risk measures, (7) yields

[TABLE]

where $\Af^{i}$ is the ambiguity set of $\rho^{i}$. Conditional risk mappings are used to measure the risk of a multistage stochastic process $\left(Z_{0},\ldots,Z_{t}\right)$ of random costs, which evolves on a scenario tree. Given a sequence $(\rho_{\mid{}0},\ldots,\rho_{\mid{}t})$ of conditional risk mappings, we define

[TABLE]

which is called a nested multistage risk measure. We define the composite risk measure at stage $t$ as

[TABLE]

If all $\rho^{i}$ are coherent risk measures, then $\bar{\rho}_{t}$ is a coherent risk measure on $\operatorname{nodes}(t)$ [1].

IV Risk-constrained risk-averse optimal control

IV-A Risk-averse optimal control problems

A risk-constrained risk-averse optimal control problem with horizon $N$ is defined via the following multistage nested formulation [1, Sec. 6.8.1]

[TABLE]

for all $t\in\N_{[0,N-1]}$. Constraints (10c) are risk constraints involving risk measures $r_{t}$ on the probability spaces $\operatorname{nodes}(t+1)$ and $r_{N}$ is a risk measure on $\operatorname{nodes}(N)$. Their role is discussed in Section IV-B. The infima in (10) are taken with respect to causal control functions $u_{t}$.

The above nested formulation amounts to minimizing the nested multistage cost $\varrho_{N-1}(\ell_{0}(x_{0},u_{0},w_{0}),\ldots,\ell_{N}(x_{N}))$ subject to the system dynamics and additional constraints [1, Sec. 6.8]. Replacing the conditional risk mappings, $\rho_{\mid{}t}$, with conditional expectations, $\E_{\mid{}t}$, results in a standard expectation-based problem [22]. Similarly, when the underlying risks are the maximum operators, we obtain a minimax problem [22]. Therefore, risk-averse problems generalize risk-neutral and minimax formulations and contain them as special cases. Moreover, the above formulation enables the stability analysis of associated model predictive control formulations [23, 24].

IV-B Risk constraints

At each stage $t\in\N_{[0,N-1]}$, let us define $q_{t}$ functions $\phi_{j,t}:\Re^{n_{x}}\times\Re^{n_{u}}\times\Re^{n_{w}}\to\Re$, $j\in\N_{[1,q_{t}]}$. At stage $N$, we also define $q_{N}$ functions $\phi_{j,N}:\Re^{n_{x}}\to\Re$, $j\in\N_{[1,q_{N}]}$. Reciting [25], our objective is to impose that “$\phi_{j,t}$ are adequately $\leq{}0$,” for $t\in\N_{[0,N]}$, in a probabilistic sense.

Let $G_{j,t}=\phi_{j,t}(x_{t},u_{t},w_{t})$ be a real-valued random quantity defined at stage $t\in\N_{[0,N-1]}$ and $G_{j,N}=\phi_{j,N}(x_{N})$. Similar to the definition of $Z^{i}$ in Sec. III-B, at every stage $t\in\N_{[0,N-1]}$ and node $i\in\operatorname{nodes}(t+1)$, we assign values $G_{j,t}=((G_{j}^{i})_{i\in\operatorname{nodes}(t+1)}$ for every $j\in\N_{[1,q_{t}]}$. Analogously, we define $G_{j,N}$ for $j\in\N_{[1,q_{N}]}$

Risk constraints may serve several purposes: (i) the average (and the entropic) value-at-risk can be used as a convex approximation of chance constraints [15]. Chance constraints of the form $\prob[G_{j,t}{}\leq{}0]\geq 1-\delta$ can be approximated by risk constraints of the form $\AVAR_{\delta}[G_{j,t}]\leq 0$ (or $\EVAR_{\delta}[G_{j,t}]\leq 0$) — in particular, $\AVAR$ offers a tight convex approximation to chance constraints [26, Sec. 4.3.3], (ii) to impose ambiguous expectation constraints, that is, constraints of the form $\E^{\mu}[G_{j,t}]\leq 0$ for all $\mu$ in a set $\mathcal{M}$ [5], and lastly, (iii) to accommodate ambiguity in chance constraints, i.e., $\prob[G_{j,t}{}\leq{}0]\geq 1-\delta$ for all $\prob\in\mathcal{M}$ [1]. Here, we study two different risk constraint formulations on scenario trees, namely, (i) stage-wise risk constraints, (ii) multistage nested risk constraints.

IV-B1 Stage-wise risk constraints

Stage-wise constraints are imposed at every stage $t\in\N_{[0,N-1]}$ as follows

[TABLE]

for $j\in\N_{[1,q_{t}]}$, where $r_{t}:\Re^{|\operatorname{nodes}(t+1)|}\to\Re$ are risk measures and $G_{j,t}\in\Re^{|\operatorname{nodes}(t+1)|}$. At $t=N$, similarly, we impose $r_{N-1}[G_{j,N}]\leq 0$ for $j\in\N_{[1,q_{N}]}$. But, such risk-based constraints do not account for how the probability distribution at stage $t$ is generated in time; indeed, the dependence on previous stages in (11) is disregarded. This can lead to certain pathological cases as we demonstrate in the following section.

IV-B2 Multistage nested risk constraints

Consider a scenario tree generated by an iid process $(w_{t})_{t\in\N_{[0,N-1]}}$ with $w_{t}\in\{0,1\}$ as the one in Fig. 3 with $\prob[w_{t}=0]=0.2$ and $\prob[w_{t}=1]=0.8$. Let functions $(\phi_{t})_{t\in\N_{[0,N]}}$ be such that $G^{i}=0$ for $i\neq 15$ and $G^{15}=10^{2}$. The (nominal) probability of node $i=15$ is $\pi^{15}=0.2^{4}=0.0016$. Suppose that the probability has been misestimated and the actual one is $\prob^{\prime}[w_{t}=0]=0.25$ and $\prob^{\prime}[w_{t}=1]=0.75$. This probability vector is within $\Af^{\mathrm{avar}}_{0.8}(\prob)$. On the other hand, we have that $\pi^{\prime{}15}=0.25^{4}=0.0039$, but the ambiguity set $\Af^{\mathrm{avar}}_{0.8}$ on $\operatorname{nodes}(4)$ contains no $\mu$ such that $\mu[\{15\}]=0.0039$. In other words, the stage-wise risk fails to describe how ambiguity may build up and propagate in time. This motivates the introduction of multistage nested risk constraints of the form

[TABLE]

Here, using the average value-at-risk with parameter $\alpha=0.8$, we compute the risk $\AVAR_{0.8}[G_{3}]\,{=}\,0.2$; the corresponding nested risk is $\overline{\AVAR}_{0.8,3}[G_{3}]\,{=}\,0.3906$. Note that nested risk constraints neither imply nor are implied by stage-wise ones.

V Tractable reformulations

V-A Conic representation of risk measures

The ambiguity set of a coherent risk measure can be written using conic inequalities, i.e., there exist matrices $E,F$ and a vector $b$, such that

[TABLE]

where $\mathcal{K}$ is a closed, convex cone and $\nu$ is an auxiliary variable. All widely used coherent risk measures can be written in this form. Tacitly, we have assumed that all admissible $\mu$ in (13) are probability vectors and the ambiguity set of $\rho$ is the following subset of $\mathcal{D}_{n}$

[TABLE]

For example, $\AVARa$ is written as in (13) with $r=0$ and E=\smallmat{I&-I1_{n}}^{{\scriptscriptstyle\top}}, b=\smallmat{\nicefrac{{1}}{{\alpha}}\pi^{{\scriptscriptstyle\top}}&01}^{{\scriptscriptstyle\top}} and $\mathcal{K}=\Re_{\geq 0}^{2n}\times\{0\}$. $\EVARa$ can also be written in the above form. Let $\mathcal{K}^{\mathrm{e}}=\operatorname{cl}\{(x,y,z)\in\Re^{3}{}\mid{}ye^{x/y}\leq z,y>0\}$ be the exponential cone. By virtue of the equivalence $x\ln(x/y)\leq t\Leftrightarrow(-t,x,y)\in\mathcal{K}^{\mathrm{e}}$, the ambiguity set $\Af^{\mathrm{evar}}_{\alpha}(\pi)$ is

[TABLE]

Lastly, note that if $\rho$ is admits a conic representation, so does the regularized risk measure $\rho^{\lambda}$, which has the ambiguity set

[TABLE]

Provided that strong duality holds — which is the case if there exist $\mu^{*}$ and $\nu^{*}$ so that $b-E\mu^{*}+F\nu^{*}\in\operatorname{ri}(\mathcal{K})$ [27, Thm. 1.4.2] — the risk measure in (13) can be written as

[TABLE]

We shall use this representation of risk measures to derive computationally tractable reformulations of optimal control problems involving risks.

V-B Decomposition of nested formulation

In this section we propose a computationally tractable reformulation of problem (10) using the following result

Theorem V.1** (Risk-infimum interchangeability)**

Let $\rho:\Re^{n}\rightarrow\Re$ be a convex risk measure and $g:\Re^{m}\ni x\mapsto(g_{1}(x),\ldots,g_{n}(x))\in\Re^{n}$ where $g_{i}:\Re^{m}\rightarrow\Re$ is an lsc, level-bounded function over a closed set $\emptyset\neq X\subseteq\Re^{m}$. Let $\inf_{x\in X}g(x){}\dfn{}\left(\inf_{x\in X}g_{1}(x),\dots,\inf_{x\in X}g_{n}(x)\right)$. Then

[TABLE]

Furthermore, if $\rho$ is strictly monotone or $\rho\circ g:\Re^{m}\to\Re$ is strictly convex over $X$, then

[TABLE]

Proof:

The proof is given in the appendix. ∎

The epigraph of a risk measure $\rho:\Re^{n}\to\Re$ is the set $\operatorname{epi}\rho{}={}\{(Y,\gamma)\in\Re^{n+1}{}\mid{}\rho[{}Y{}]{}\leq{}\gamma\}$. When $\rho$ is a coherent risk measure given by (14), its epigraph is the set

[TABLE]

Then, for example, stage-wise risk constraints (11) are equivalent to $r_{t}(\inf_{G_{j,t}\leq\eta_{j,t+1}}\eta_{j,t+1})\leq 0$ for a random variable $\eta_{t+1}\in\Re^{|\operatorname{nodes}(t+1)|}$. Using Thm. V.1, we have that the risk constraints (11) are equivalent to the existence of $\eta_{j,t+1}$ such that $G_{j,t}\leq\eta_{j,t+1}$ and $(\eta_{t+1},0)\in\operatorname{epi}r_{t}$.

We shall now derive the epigraph of nested risk measures. To that end, we first define the epigraph of a conditional risk mapping $\operatorname{epi}\rho_{\mid{}t}{}={}\{(Y_{t+1},Y_{t})\in\Re^{|\operatorname{nodes}(t+1)|+|\operatorname{nodes}(t)|}{}\mid{}\rho_{\mid{}t}[{}Y_{t+1}{}]{}\leq{}Y_{t}\}$ which is the Cartesian product of the epigraphs of its constituent risk measures

[TABLE]

Proposition V.2** (Nested risk epigraph)**

Let $(\rho_{\mid{}0},\ldots,\rho_{\mid{}t})$ be a sequence of coherent conditional risk mappings. Let $\bar{\rho}_{t}$ be the corresponding nested risk measure. Its epigraph is

[TABLE]

Proof:

The proof is given in the appendix. ∎

Using Prop. V.2, we may write (12) in the form $(G_{j,t},0)\in\operatorname{epi}\bar{r}_{t}$. Risk constraints, both stage-wise and nested, can be cast as conic constraints. By virtue of the interchangeability property in Theorem V.1, problem (10) is written as

[TABLE]

subject to (10b)–(10d). Similarly, this is equivalent to

[TABLE]

subject to (10b)–(10d), where $\tau_{t}\in\Re^{|\operatorname{nodes}(t)|}$ and $s_{N}\in\Re^{|\operatorname{nodes}(N)|}$. Starting by epigraphically relaxing the innermost term, $\rho_{\mid{}N-1}[\tau_{N}+s_{N}]=\inf\{s_{N-1}\mid(\tau_{N}+s_{N},s_{N-1})\in\operatorname{epi}\rho_{\mid{}N-1}\}$, proceeding backwards, employing Thm. V.1 and using the dual conic representation of risk measures, we obtain the following formulation

[TABLE]

In particular, if each $\rho^{i}$ is a conic risk measure which is described by the tuple $E^{i},F^{i},b^{i},\mathcal{K}^{i}$ then the above optimization problem boils down to

[TABLE]

Suppose that the problem is subject to stage-wise risk constraints of the form (11) at stage $t$ with a conic risk measure $r_{t}$ described by the tuple $(\bar{E}_{t},\bar{F}_{t},\bar{b}_{t},\bar{\mathcal{K}}_{t})$. For notational convenience, we drop the index $j$.

[TABLE]

for $i\in\operatorname{nodes}(t)$, $i_{+}\in\operatorname{child}(i)$. We have here introduced the additional variables $\eta_{t}\in\Re^{|\operatorname{nodes}(t+1)|}$ and $\bar{y}_{t}$.

Similarly, suppose that the problem is subject to multistage nested risk constraints at stage $t$ of the form (12) where the multistage risk is given by conic risk measures $r^{i}$ described by the tuples $(\tilde{E}^{i},\tilde{F}^{i},\tilde{b}^{i},\tilde{\mathcal{K}}^{i})$. Then, (12) leads to the following constraints

[TABLE]

for $i^{\prime}{\in}\operatorname{nodes}(t)$, $i^{\prime}_{+}{\in}\operatorname{child}(i^{\prime})$, $i{\in}\operatorname{nodes}(t^{\prime})$, $t^{\prime}{\in}\N_{[1,t]}$.

In all cases, the number of decision variables and constraints increases linearly with the total number of nodes. Although nested risk constraints are translated to more constraints than their stage-wise counterparts, the associated complexity is of the same order of magnitude (see Section VI for computation times).

When the system dynamics is linear (or affine) and functions $\ell_{t}$ and $\phi_{t}$ are convex in $x$ and $u$, then (19) is a convex conic problem which can be solved very efficiently with solvers such as MOSEK [28], SuperSCS [29] and more.

Problems (10) and (19) are equivalent in the sense that the optimal values of the objective function at the solution are the same. If all involved risk measures are strictly monotone, then the respective sets of minimizers are equal.

An important property that allows to establish a link between (10) and (19) is that of time consistency of a policy $(u_{0},\ldots,u_{N-1})$; a policy is called time consistent if for every $t=1,\ldots,N-1$, the tail $(u_{t},\ldots,u_{N-1})$ is optimal conditional on $(x_{0},\ldots,x_{t})$ [1]. Clearly, all solutions of (10) are time consistent. According to [18, Thm. 2], all time consistent solutions of (19) are optimal for (10).

In control applications on Markovian switching systems, such as [24], problem formulations akin to (19) are employed in a receding horizon fashion: a multistage risk-averse problem is solved at each time instant and the first control action is applied to the dynamical system. The fact that not all policies are time consistent does not compromise the stability properties of the closed loop.

VI Illustrative example

Suppose $(w_{t})_{t}$ is governed by a stopped Markov process, that is $w_{t}{}={}W_{\min(t,t_{0})}$, where $(W_{t})_{t}$ is a Markov process with $m=4$ modes. Suppose the system evolves as a Markov jump linear system driven by $(w_{t})_{t}$, that is, $f(x,u,w)=A_{w}x+B_{w}u$, the stage cost is given by $\ell_{t}(x,u,w)=x^{{\scriptscriptstyle\top}}Q_{w}x+u^{{\scriptscriptstyle\top}}R_{w}u$, the prediction horizon is $N=7$. The system dimensions are $n_{x}=2$ and $n_{u}=1$. Matrices $A_{w}$, $B_{w}$, $Q_{w}$ and $R_{w}$ were selected randomly. The input constraints $-10\leq u_{t}\leq 10$ are imposed on the control actions and suppose, for now, that no risk constraints are imposed.

Consider the cumulative probability distribution of the total cost $\sum_{t=0}^{N-1}\ell_{t}(x_{t},u_{t},w_{t})+\ell_{N}(x_{N})$ in Fig. 4. The worst-case cost is minimal for $\alpha=0$ at the expense of a higher cost when moving away from the extremes. By contrast, for $\alpha=1$, the expected cost is minimal, yet high costs may occur with low probability. Intermediate values of $\alpha$ result in a trade-off between the two, effectively determining the extent to which the right tail of the distribution of the cost is compressed.

Next, consider the stage constraints function $\phi_{t}(x,u,w)=\|x\|^{2}-c$ with $c=0.5$. We use $\AVAR_{0.5}$ in the cost and impose the stage-wise risk constraint $\AVARa(\phi_{t}(x_{t},u_{t},w_{t}))\leq 0$ at all stages $t\in\N_{[N-4,N-1]}$. For the robust case ($\alpha=0$), there was no feasible solution. As $\alpha$ increases, Fig. 5 shows that constraint violations occur in larger fractions of the realisations. Also note that since $\AVARa$ bounds the $(1-\alpha)$-quantile function, it is guaranteed that $\prob[\|x_{t}\|\leq c]\geq 1-\alpha$.

Fig. 6 shows the complexity of the optimization problem with respect to the number of scenarios. We fixed $N=12$ and controlled the number of scenarios and nodes with the branching horizon $t_{0}$. Although nested risk constraints increase the problem size, the asymptotic complexity remains linear in the number of nodes. Moreover, for both constraint types, problems using $\AVAR$ with up to 200 scenarios can still be solved in well under a second. Since MOSEK V. 8 [28] does not support exponential constraints, it cannot be used to solve problems involving $\EVAR$. For that reason, we resort to SuperSCS [29]. As shown in Fig. 6, $\EVAR$-based problems are solved at a significantly higher runtime.

VII Conclusions

We presented a decomposition methodology for nested conditional risk mappings in multistage risk-averse optimal control problems. In the common case where the system dynamics is linear and the state and input constraints are convex, the original multistage nested problem is cast as a conic problem, which can be solved very efficiently and is suitable for real-time embedded applications, provided that the tree contains a moderate number of nodes. The proposed approach hinges on the convex dual formulation of conic risk measures. Future work will focus on tailored numerical methods to exploit the tree structure to solve such problems fast, efficiently and using parallelization (e.g., on GPUs [30, 31]).