Relaxing The Hamilton Jacobi Bellman Equation To Construct Inner And Outer Bounds On Reachable Sets

TL;DR

This paper introduces a method to approximate both inner and outer bounds of reachable sets for polynomial systems by relaxing the Hamilton-Jacobi-Bellman equation, using polynomial bounds on the value function.

Contribution

It presents a novel approach to construct provable bounds on reachable sets via polynomial sub- and super-value functions solving a relaxed HJB PDE.

Findings

Polynomial bounds effectively approximate reachable sets.

Inner and outer bounds have negligible Hausdorff distance at low polynomial degree.

Method applicable to systems with polynomial dynamics and semialgebraic constraints.

Abstract

We consider the problem of overbounding and underbounding both the backward and forward reachable set for a given polynomial vector field, nonlinear in both state and input, with a given semialgebriac set of initial conditions and with inputs constrained pointwise to lie in a semialgebraic set. Specifically, we represent the forward reachable set using the value function which gives the optimal cost to go of an optimal control problems and if smooth satisfies the Hamilton-Jacobi- Bellman PDE. We then show that there exist polynomial upper and lower bounds to this value function and furthermore, these polynomial sub-value and super-value functions provide provable upper and lower bounds to the forward reachable set. Finally, by minimizing the distance between these sub-value and super-value functions in the L1-norm, we are able to construct inner and outer bounds for the reachable set and show numerically on several examples that for relatively small degree, the Hausdorff distance between these bounds is negligible.

Figures

Equations

$$\dot{x}(t)=f(x(t),\mathbf{u}(t)),\mathbf{u}(t)\in Y,x(0)\in X_0\in {\mathbb{R}}^n,$$

$$U_Y:=\{\mathbf{u}:\mathbb{R}\to {\mathbb{R}}^m:\mathbf{u}(t)\in Y\text{ for all }t\in (-\infty ,\infty )\}.$$

$$\frac{\partial h(x,t,\mathbf{u})}{\partial t}=f(h(x,t,\mathbf{u}),\mathbf{u}(t))$$

$$h(x,0,\mathbf{u})=x,$$

$${\varphi }_{-f}(x,-t,\hat{\mathbf{u}})={\varphi }_f(x,t,\mathbf{u}).$$

$${\varphi }_f(x,t+s,\mathbf{u})={\varphi }_f({\varphi }_f(x,s,\mathbf{u}),t,{\tilde{\mathbf{u}}}_s)$$

$$\frac{\partial {\varphi }_{-f}(x,t,\mathbf{u})}{\partial t}=-f({\varphi }_{-f}(x,t,\mathbf{u}),\mathbf{u}(t))\text{ and }{\varphi }_{-f}(x,0,\mathbf{u})=x.$$

$$\frac{\partial h_1(x,t,\mathbf{u})}{\partial t}$$

$$=f({\varphi }_{-f}(x,s,\hat{\mathbf{u}}),\hat{\mathbf{u}}(s))=f({\varphi }_{-f}(x,-t,\hat{\mathbf{u}}),\mathbf{u}(t))$$

$$=f(h_1(x,t,\mathbf{u}),\mathbf{u}(t)),$$

$$h_2^s(x,t,\mathbf{u}):=\{\begin{array}{l}{\varphi }_f(x,t,\mathbf{u})\text{ for }-T\le t\le s\\ {\varphi }_f({\varphi }_f(x,s,\mathbf{u}),t-s,{\tilde{\mathbf{u}}}_s)\text{ for }t\in (s,T].\end{array}$$

$$\frac{\partial h_2^s(x,t,\mathbf{u})}{\partial t}=\frac{\partial {\varphi }_f({\varphi }_f(x,s,\mathbf{u}),t-s,{\tilde{\mathbf{u}}}_s)}{\partial t}$$

$$=\frac{\partial {\varphi }_f({\varphi }_f(x,s,\mathbf{u}),k,{\tilde{\mathbf{u}}}_s)}{\partial k}=f({\varphi }_f({\varphi }_f(x,s,\mathbf{u}),k,{\tilde{\mathbf{u}}}_s),{\tilde{\mathbf{u}}}_s(k))$$

$$=f({\varphi }_f({\varphi }_f(x,s,\mathbf{u}),t-s,{\tilde{\mathbf{u}}}_s),{\tilde{\mathbf{u}}}_s(t-s))=f(h_2(x,t,\mathbf{u}),\mathbf{u}(t)),$$

$$F{R}_f(X_0,Y,S):=\{y\in {\mathbb{R}}^n:$$

$$\text{such that }{\varphi }_f(x,t,\mathbf{u})=y\}.$$

$$V^{\ast }(x,t)=$$

$$\underset{\mathbf{u}\in U_Y}{\inf }\left\{{\int }_t^{T}c({\varphi }_f(x,s-t,\mathbf{u}),\mathbf{u}(s),s)ds+g({\varphi }_f(x,T-t,\mathbf{u}))\right\}.$$

$${\nabla }_{t}V(x,t)+\underset{u\in Y}{\inf }\left\{c(x,u,t)+{\nabla }_{x}V(x,t{)}^{T}f(x,u)\right\}=0$$

$$V(x,T)=g(x).$$

$$J(x,t)\le V^{\ast }(x,t)\forall t\in [0,T]\text{ and }x\in F{R}_f(X_0,Y,\{t\}),$$

$$J(x,t)\ge V^{\ast }(x,t)\forall t\in [0,T]\text{ and }x\in F{R}_f(X_0,Y,\{t\}),$$

$$B{R}_f(X_0,Y,S):=\{y\in {\mathbb{R}}^n:$$

$$\text{such that }{\varphi }_f(y,t,\mathbf{u})=x\}.$$

$${\varphi }_{-f}(x,T,\mathbf{u})=y.$$

$${\varphi }_f(y,T,\tilde{\mathbf{u}})$$

$$={\varphi }_f(x,T-T,\hat{\mathbf{u}})=x,$$

$${\varphi }_f(y,T,\mathbf{u})=x.$$

$${\varphi }_{-f}(x,T,\mathbf{w})$$

$$={\varphi }_f(y,T-T,\mathbf{u})=y,$$

$$B{R}_f(X_0,Y,\{T\})=\{x\in X:V^{\ast }(x,0)\le 1\}.$$

$$V^{\ast }(x,t)=\underset{\mathbf{u}\in U_Y}{\inf }g({\varphi }_f(x,T-t,\mathbf{u})).$$

$$V^{\ast }(y_0,0)$$

$$g(x_0)=g({\varphi }_f(y_0,T,{\mathbf{u}}_0))=\underset{\mathbf{u}\in U_Y}{\inf }g({\varphi }_f(y_0,T,\mathbf{u}))=V^{\ast }(y_0,0)\le 1,$$

$$\{x\in X:V_u(x,0)\le 1\}$$

$$B{R}_f(X_0,Y,\{T\})$$

Full text

I Introduction

The reachable set of an ODE is the set of coordinates that can be reached by the solution map, defined in Assumption 1, at some fixed time and starting in some set of initial conditions. The computation of reachable sets is important for certifying solution maps remain in “safety regions”; regions of the state space that are deemed to have low risks of system failure. Historic examples of solution maps transitioning outside “safe regions” include: two of the four reaction wheels on the Kepler Space telescope failing, analyzed in [1]; and the disturbing lateral vibrations of the Millennium footbridge over the River Thames in London on opening day, analyzed in [2] and [3].

In this paper we show the reachable set of an ODE, subject to pointwise bounded inputs, is the sublevel set of the “value function” (optimal cost to go function) associated with a one player optimal control problem. This result can be thought of as the analogous result to [4]; where it was shown the reachable set of an ODE, subjected to two sets of adversarially opposed input parameters, is the sublevel set of the “value function” associated with a two player optimal control problem.

It is known that if the “value function” of a one player optimal control problem is smooth then it satisfies the Hamilton Jacobi Bellman (HJB) Partial Differential Equation (PDE) [5]. In this paper we show that relaxing the HJB PDE to a dissipation inequality allows for the construction of upper and lower bounds of the ”value function”; we call super-value and sub-value functions respectively. We futhermore give sufficient conditions for the existence of polynomial super-value and sub-value functions. Moreover, it is shown that the sublevel set of sub-value and super-value functions construct provable upper and under bounds of reachable sets respectively.

The HJB PDE may not always have a solution in the classical sense. A generalized solution concept, called the viscosity solution, was developed in [6]. Discretization methods, such as those in [7] [8], are typically used to approximate the viscosity solution. However, such methods cannot guarantee that the approximate viscosity solution is an upper or lower bound to the true ”value function”. Alternatively, we propose a Sum-of-Squares (SOS) optimization problem that is solved by the polynomial sub-value and super-value functions with minimum $L_{1}$ distance.

Our approach to finding sub- and super-solutions to the HJB PDE is similar to [9] and [10]. In [9] SOS was used to find a sub-value function for optimal control problems with discrete-time dynamics; whereas we consider continuous-time dynamics. In [10] SOS was used to find sub-value and super-value functions for optimal control problems with quadratic costs and continuous-time synamics governed by ODE’s affine in the input variable. Our approach allows us to construct sub-value and super-value functions for more general optimal control problems with polynomial costs and continuous time varying processes governed by ODE’s nonlinear in the input variable. Moreover, we give sufficient conditions on the existence of polynomial sub-value and super-value functions and show how these functions can be used for reachable set estimation.

We numerically demonstrate that solving our proposed SOS optimization problem can give tight approximations of reachable sets. Unlike alternative approaches to reachable set analysis, [4] [11] [12], our reachable set approximations can be proved to overbound or underbound the reachable set.

An alternative approach to reachable set approximation is found in [13] [14] [15] [16] where dissipation like inequalities are solved using SOS programing to find a function whose sublevel set contains the reachable set. It is shown in this paper such dissipation inequalities are actually relaxations of the HJB PDE and thus solved by sub-value functions. In this context, our sufficient conditions for the existence of polynomial sub-value functions for optimal control problems can be viewed as feasibility conditions for the SOS optimization problems found [13] [14] [15] [16].

The paper is organized as follows. Background material on ODE’s is given in Section III. In Section IV optimal control theory is presented. In Section V we construct an optimal control problem with value function that can characterize the reachable set exactly. In Section VI we show how relaxing the HJB PDE allows us to derive dissipation inequalities that are solved by sub-value and super-vale functions. In Section VII an SOS optimization is proposed that minimizes the $L_{1}$ norm of the distance between the sub-value and super-value function. The conclusion is given in Section VIII.

II Notation

We denote a ball with radius $R>0$ centered at the origin by $B_{R}=\{x\in\mathbb{R}^{n}:x^{T}x<R^{2}\}$. For $x\in\mathbb{R}^{n}$ we denote $||x||_{\infty}=\max_{1\leq i\leq n}|x_{i}|$. For short hand we denote the partial derivative $D^{\alpha}f(x):=\Pi_{i=1}^{n}\frac{\partial^{\alpha_{i}}f}{\partial x_{i}^{\alpha_{i}}}(x)$ for $\alpha\in\mathbb{N}^{n}$. Let $C(\Omega)$ be the Banach space of scalar continuous functions with domain $\Omega\subset\mathbb{R}^{n}$. For $f\in C(\Omega)$ we define the norms $||f||_{\infty,\Omega}:=\sup_{x\in\Omega}||f(x)||_{\infty}$ and $||f||_{1,\Omega}=\int_{\Omega}|f(x)|dx$. We denote the set of differentiable functions by $C^{i}(\Omega):=\{f\in C(\omega):D^{\alpha}f\in C(\Omega)\text{ }\forall\alpha\in\mathbb{N}^{n}\text{ such that }\sum_{j=1}^{n}\alpha_{j}\leq i\}$. For $V\in C^{1}(\mathbb{R}^{n}\times\mathbb{R})$ we denote $\nabla_{x}V:=(\frac{\partial V}{\partial x_{1}},....,\frac{\partial V}{\partial x_{1}})$ and $\nabla_{t}V=\frac{\partial V}{\partial x_{n+1}}$. For $d\in\mathbb{N}$ and $x\in\mathbb{R}^{n}$ we denote $z_{d}(x)$ to be the vector of monomial basis in $n$-dimensions with maximum degree $d\in\mathbb{N}$. We denote the space of scalar valued polynomials $p:\Omega\to\mathbb{R}$ with degree at most $d\in\mathbb{N}$ by $\mathcal{P}_{d}[\Omega]$. We say $p\in\mathcal{P}_{d}[\mathbb{R}^{n}]$ is Sum-of-Squares (SOS) if there exists $p_{i}\in\mathcal{P}_{d}[\mathbb{R}^{n}]$ such that $p(x)=\sum_{i=1}^{k}(p_{i}(x))^{2}$. We denote $\sum_{SOS}$ to be the set of SOS polynomials.

III Background: Differential Equations

We consider nonlinear Ordinary Differential Equations (ODE’s) of the form

[TABLE]

where $f:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{n}$; $u:\mathbb{R}\to\mathbb{R}^{m}$ is the input; and $Y\subset\mathbb{R}^{m}$ and $X_{0}\subset\mathbb{R}^{n}$ are compact sets representing constraints on the inputs and initial conditions.

To define the solution map we define the set of pointwise-admissible input signals as

[TABLE]

For a given set of admissible inputs, we constrain $f$, in the following definition, to admit a continuously-differentiable solution map.

Definition 1** (Constraint on Admissibility of $f$)**

For given $Y\subset\mathbb{R}^{m}$ we say $f\in\mathcal{F}_{Y}$ if

$f_{i}\in C^{1}(\mathbb{R}^{n})$* for all $i\in\{1,...,n\}$.* 2. 2.

For any $T>0$, there exists a function $h:\mathbb{R}^{n}\times\mathbb{R}\times U_{Y}\to\mathbb{R}^{n}$, where for any $\mathbf{u}\in U_{Y}$ we have $h_{i}(\cdot,\cdot,\mathbf{u})\in C^{1}(\mathbb{R}^{n}\times[-T,T])$ for all $i\in\{1,...,n\}$, and

[TABLE]

for all $x\in\mathbb{R}^{n}$, $t\in[-T,T]$ and $\mathbf{u}\in U_{Y}$. 3. 3.

The function $h$ that satisfies (2) is unique.

Since for each $f\in\mathcal{F}_{Y}$ the associated function that satisfies (2) is unique we will denote this function by $\phi_{f}$ throughout the paper.

Lemma 1

Let $Y\subset\mathbb{R}^{m}$ be a compact set, $f,-f\in\mathcal{F}_{Y}$ and $T\in\mathbb{R}^{+}$.

1. (A)

For $\mathbf{u}\in U_{Y}$ define $\hat{\mathbf{u}}(t)=\mathbf{u}(-t)$, then $\forall x\in\mathbb{R}^{n},t\in[-T,T]$

[TABLE] 2. (B)

For $s\in[-T,T]$ and $\mathbf{u}\in U_{Y}$ define $\tilde{\mathbf{u}}_{s}(t)=\mathbf{u}(t+s)$, then $\forall x\in\mathbb{R}^{n},t\in[-T-s,T-s]\cap[-T,T]$

[TABLE]

Proof:

Proving (3) in Statement (A): As $-f\in\mathcal{F}_{Y}$ we have for all $x\in\mathbb{R}^{n}$, $t\in[-T,T]$, and $\mathbf{u}\in U_{Y}$

[TABLE]

Now, letting $h_{1}(x,t,\mathbf{u})=\phi_{-f}(x,-t,\hat{\mathbf{u}})$, for $x\in\mathbb{R}^{n}$, $t\in[-T,T]$, and $\mathbf{u}\in U_{Y}$ the following holds

[TABLE]

where to get the second equality we use $s=-t$, so $ds=-dt$; to get the third equality (5) was used; to get the fourth equality the substitution $s=-t$ was again applied, noting $\hat{\mathbf{u}}(-t)=\mathbf{u}(t)$. Moreover, as $h_{1}(x,0,\mathbf{u})=\phi_{-f}(x,0,\hat{\mathbf{u}})=x$, by (5), it follows $h_{1}$ satisfies (2) and therefore, due to the uniqueness of $\phi_{f}$, (3) must follow.

Proving (4) in Statement (B): For fixed $s\in[-T,T]$ let us consider the following function

[TABLE]

We prove (4) by showing $h_{2}$ satisfies (2) and using the uniqueness properties of $\phi_{f}$. Firstly it is clear $h_{2}^{s}(x,0,\mathbf{u})=\phi_{f}(x,0,\mathbf{u})=x$, and $h_{2}^{s}$ satisfies (2) for all $x\in\mathbb{R}^{n}$, $t\in[-T,s]$ and $\mathbf{u}\in U_{Y}$. Now for all $x\in\mathbb{R}^{n}$, $t\in(s,T]$ and $\mathbf{u}\in U_{Y}$

[TABLE]

where the second equality follows from using $k=t-s$ so $dk=dt$; the third equality follows by (2); the fourth equality follows from applying $k=t-s$ again; the fifth equality follows as $\tilde{\mathbf{u}}_{s}(t-s)=\mathbf{u}(t)$.

Thus by the uniqueness of $\phi_{f}$ it follows $\phi_{f}(x,t+s,\mathbf{u})=h_{2}(x,t+s,\mathbf{u})=\phi_{f}(\phi_{f}(x,s,\mathbf{u}),t,\tilde{\mathbf{u}}_{s})$, therefore showing (4).

∎

For a given $X_{0}\subset\mathbb{R}^{n}$, $Y\subset\mathbb{R}^{n}$ and $f\in\mathcal{F}_{Y}$, we next define the forward reachable set as follows.

Definition 2

For $X_{0}\subset\mathbb{R}^{n}$, $Y\subset\mathbb{R}^{m}$, $f\in\mathcal{F}_{Y}$ and $S\subset\mathbb{R}^{+}$, let

[TABLE]

In following sections, $S$ is of the form either $\{T\}$ or $[0,T]$.

IV Finite Time Optimal Control Problems

An optimal control problem with finite time horizon is a tuple $\{c,g,f,X_{0},Y,T\}$ where $c:\mathbb{R}^{n}\times\mathbb{R}\times\mathbb{R}\to\mathbb{R}$ is the running cost; $g:\mathbb{R}^{n}\to\mathbb{R}$ is the terminal cost; $f\in\mathcal{F}_{Y}$; $X_{0}\subset\mathbb{R}^{n}$ is the set of initial conditions; $Y\subset\mathbb{R}^{m}$ is a compact input set; and $T$ is the final time. For each optimal control problem we can next define the value function that intuitively describes the optimal ”cost to go”.

Definition 3

For given $X_{0}\subset\mathbb{R}^{n}$; $Y\subset\mathbb{R}^{m}$; $T>0$; $c:\mathbb{R}^{n}\times\mathbb{R}\times\mathbb{R}\to\mathbb{R}$; $g:\mathbb{R}^{n}\to\mathbb{R}$; $f\in\mathcal{F}_{Y}$ we say $V^{*}:\mathbb{R}^{n}\times\mathbb{R}\to\mathbb{R}$ is a value function of the tuple $\{c,g,f,X_{0},Y,T\}$ if for $(x,t)\in(FR_{f}(X_{0},Y,\{t\}),t)$, where $t\in[0,T]$, the following holds

[TABLE]

A sufficient condition for $V^{*}$ to be a value function for the tuple $\{c,g,f,X_{0},Y,T\}$ is for $V^{*}$ to satisfy the Hamilton Jacobi Bellman (HJB) PDE.

Proposition 1

For given $X_{0}\subset\mathbb{R}^{n}$, $Y\subset\mathbb{R}^{m}$, $g:\mathbb{R}^{n}\to\mathbb{R}$, $c:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}\to\mathbb{R}$, $f\in\mathcal{F}_{Y}$, $T>0$, suppose there exists a differentiable function $V\in C^{1}(\mathbb{R}^{n}\times\mathbb{R})$ such that the following holds for $(x,t)\in FR_{f}(X_{0},Y,[0,T])\times[0,T]$

[TABLE]

*Then $V$ is the value function of the optimal control problem $\{c,g,f,X_{0},Y,T\}$. *

Proof:

Follows by Proposition 3.2.1 from [5] where the domain of the value function is restricted to $(x,t)\in(FR_{f}(X_{0},Y,\{t\}),t)$. ∎

Definition 4

We say the function $J:\mathbb{R}^{n}\times\mathbb{R}\to\mathbb{R}$ is a sub-value function to the finite time horizon optimal control problem $\{c,g,f,X_{0},Y,T\}$ if we have

[TABLE]

where $V^{*}$ is the value function of $\{c,g,f,X_{0},Y,T\}$. Moreover if instead $J$ satisfies

[TABLE]

we say $J$ is a super-value function to $\{c,g,f,X_{0},Y,T\}$.

V How Sublevel Sets Of Value Functions Can Describe Reachable Sets

In this section we construct a finite time horizon optimal control problem with associate value function whose sublevel sets can construct the reachable set of a system. We then show how the sublevel sets of the sub-value and super-value functions over- and under-bound the reachable set.

Analogous to Definition 2 we now define the backward reachable set and show how it is related to the forward reachable set in Lemma 2.

Definition 5

For $X_{0}\subset\mathbb{R}^{n}$, $Y\subset\mathbb{R}^{m}$, $f\in\mathcal{F}_{Y}$ and $S\subset\mathbb{R}^{+}$, let

[TABLE]

In the next Lemma we give a relationship between the backward reachable set and forward reachable set. This relationship shows finding the set $FR_{f}(X_{0},Y,\{T\})$ is equivalent to finding the set $BR_{-f}(X_{0},Y,\{T\})$. Therefore for the rest of this paper we concentrate on developing methods to bound the backward reachable set. However, for numerical implementation we will change the sign of the vector field to allow for the calculation of forward reachable set bounds.

Lemma 2

*Suppose $Y\subset\mathbb{R}^{m}$, $f\in\mathcal{F}_{Y}$ is such that $-f\in\mathcal{F}_{Y}$, and $T\in\mathbb{R}^{+}$. Then $FR_{-f}(X_{0},Y,\{T\})=BR_{f}(X_{0},Y,\{T\}).$ *

Proof:

We first show $FR_{-f}(X_{0},Y,\{T\})\subseteq BR_{f}(X_{0},Y,\{T\})$. For $y\in FR_{-f}(X_{0},Y,\{T\})$ there exists $x\in X_{0}$ and $\mathbf{u}\in U_{Y}$ such that

[TABLE]

If we denote $\hat{\mathbf{u}}(t)=\mathbf{u}(-t)$ and $\tilde{\mathbf{u}}(t)=\hat{\mathbf{u}}(t-T)$, it now follows

[TABLE]

where the first equality follows by (8), the second equality by (3), and the third equality follows by (4). Thus we deduce from (9) $y\in BR_{f}(X_{0},Y,\{T\})$.

We next show $BR_{f}(X_{0},Y,\{T\})\subseteq FR_{-f}(X_{0},Y,\{T\})$. For $y\in BR_{f}(X_{0},Y,\{T\})$ there exists $x\in X_{0}$ and $\mathbf{u}\in U_{Y}$ such that

[TABLE]

Let us denote $\mathbf{w}(t)=u(t+T)$, $\hat{\mathbf{w}}(t)=\mathbf{w}(-t)$ then it now follows

[TABLE]

where the first equality follows (3), the second equality by (10), and the third equality by (4). Thus we deduce $y\in FR_{-f}(X_{0},Y,\{T\})$. ∎

Theorem 1

Given $Y\subset\mathbb{R}^{m}$, $f\in\mathcal{F}_{Y}$ and $g:\mathbb{R}^{n}\to\mathbb{R}$, let $X_{0}=\{x\in\mathbb{R}^{n}:g(x)\leq 1\}$ and $X\subset\mathbb{R}^{n}$ be such that $BR_{f}(X_{0},Y,\{T\})\subseteq X$. Now suppose $V^{*}:\mathbb{R}^{n}\times\mathbb{R}\to\mathbb{R}$ is a value function for $\{0,g,f,X,Y,T\}$, then

[TABLE]

Proof:

As $V^{*}$ is a value function to $\{0,g,f,X,Y,T\}$ it follows for all $t\in[0,T]$ and $x\in FR_{f}(X,Y,\{t\})$

[TABLE]

For $y_{0}\in BR_{f}(X_{0},Y,\{T\})\subseteq X$ there exists $x_{0}\in X_{0}$ and $\mathbf{u}_{0}\in U_{Y}$ such that $\phi_{f}(y_{0},T,\mathbf{u}_{0})=x_{0}$. Thus it follows

[TABLE]

where the first equality follows as $y_{0}\in X$ so (12) holds. Therefore $y_{0}\in\{x\in X:V^{*}(x,0)\leq 1\}$. Hence $BR_{f}(X_{0},Y,\{T\})\subseteq\{x\in X:V^{*}(x,0)\leq 1\}$.

Now suppose $y_{0}\in\{x\in X:V^{*}(x,0)\leq 1\}$. Then if $\mathbf{u}_{0}:=\arg\inf_{\mathbf{u}\in U_{Y}}g(\phi_{f}(y_{0},T,\mathbf{u}))$, let $x_{0}:=\phi_{f}(y_{0},T,\mathbf{u}_{0})$. It follows

[TABLE]

where the third equality follows because $y_{0}\in X$ so (12) holds. Hence $x_{0}\in X_{0}$. Therefore $y_{0}\in BR_{f}(X_{0},Y,\{T\})$. Thus $\{x\in X:V^{*}(x,0)\leq 1\}\subseteq BR_{f}(X_{0},Y,\{T\})$. ∎

We next show how sub-value and super-value functions, defined in Definition 4, can can outer bound and inner bound reachable sets.

Lemma 3

Given $Y\subset\mathbb{R}^{m}$, $f\in\mathcal{F}_{Y}$ and $g:\mathbb{R}^{n}\to\mathbb{R}$, let $X_{0}=\{x\in\mathbb{R}^{n}:g(x)\leq 1\}$ and $X\subset\mathbb{R}^{n}$ be such that $BR_{f}(X_{0},Y,\{T\})\subseteq X$. Suppose $V_{l}$ and $V_{u}$ are sub-value and super-value functions to the optimal control problem $\{0,g,f,X,Y,T\}$. Then

[TABLE]

Proof:

Since $V_{l}$ and $V_{u}$ are sub-value and super-value functions to the optimal control problem $\{0,g,f,X,Y,T\}$ it follows $\forall t\in[0,T]\text{ and }x\in FR_{f}(X,Y,\{t\})$

[TABLE]

where $V^{*}$ is the value function to $\{0,g,f,X,Y,T\}$.

By (14) it follows

[TABLE]

Moreover by Theorem 1 we have

[TABLE]

Thus (15) together with (16) proves the set containments given in (13). ∎

VI Dissipation Inequalities For Sub-Value and Super-Value Functions

We now propose dissipation inequalities and show, using a novel proof, that if a differentiable function satisfies such inequalities then it must be a sub-value or super-value function associated with an optimal control problem. The dissipation inequalities are found by relaxing the HJB PDE to an inequality. A similar result is found in Theorem 3.3, from [6], for a class of PDE’s that include the HJB PDE. However in [6] a futher property, the candidate sub-value function is less than or equal to the candidate super-value function on the boundary of some compact set, is required to hold before such functions can be verified as sub-value and super-value functions.

Proposition 2

For given $T>0$, compact $Y\subset\mathbb{R}^{m}$, $g\in C^{1}(\mathbb{R}^{n})$, $c\in C^{1}(\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R})$, $f\in\mathcal{F}_{Y}$. Suppose $X_{c}\subseteq\mathbb{R}^{n}$ is such that $FR_{f}(X_{0},Y,[0,T])\subseteq X_{c}$ and $J\in C^{1}(\mathbb{R}^{n}\times\mathbb{R})$ satisfies the following $\forall x\in X_{c},u\in Y,t\in[0,T]$

[TABLE]

Then $J$ is a sub-value function to the optimal control problem $\{c,g,f,X_{0},Y,T\}$.

Alternatively if $J$ satisfies the following $\forall x\in X_{c},u\in Y,t\in[0,T]$

[TABLE]

Then $J$ is a super-value function to $\{c,g,f,X_{0},Y,T\}$.

Proof:

Let us denote the left hand side of Inequality (17) by,

[TABLE]

As $Y$ is compact and the functions $c$ and $f$ are both differentiable we may define $\tilde{L}(x,t):=\inf_{u\in Y}L(x,t,u)$. Moreover we deduce from Inequality (17) that $\tilde{L}(x,t)\geq 0$ for all $x\in X_{c}$ and $t\in[0,T]$. Now from the construction of the function $\tilde{L}$ it is clear $J$ satisfies the following equation for any $x\in X_{c},u\in Y,t\in[0,T]$

[TABLE]

If we consider the optimal control problem $\{\tilde{c},\tilde{g},f,X_{0},Y,T\}$, where $\tilde{c}(x,u,t)=c(x,u,t)-\tilde{L}(x,t)$ and $\tilde{g}(x)=J(x,T)$, as (21) holds $\forall x\in FR_{f}(X_{0},Y,[0,T])\subseteq X_{c}$ and $t\in[0,T]$ it follows by Proposition 1 $J$ is a value function for $\{\tilde{c},\tilde{g},f,X_{0},Y,T\}$. It now follows for any $t\in[0,T]$ and $x\in FR_{f}(X_{0},Y,\{t\})$ we have

[TABLE]

where $V^{*}$ is a value function of $\{c,g,f,X_{0},Y,T\}$, and the inequality follows from the fact $\tilde{L}(x,t)\geq 0$ for all $x\in X_{c}$ and $t\in[0,T]$, thus implying $\tilde{L}(\phi_{f}(x,s-t,\mathbf{u}),s)\geq 0$ for all $x\in FR_{f}(x_{0},Y,\{t\})$ and $s\in[t,T]$; and the fact $J(x,T)\leq g(x)$ for all $x\in X_{c}$, thus implying ${J}((\phi_{f}(x,T-t,\mathbf{u}),T)\leq g(\phi_{f}(x,T-t,\mathbf{u}))$ for any $x\in FR_{f}(x_{0},Y,\{t\})$. Therefore it is clear from (22) that $J$ is a sub-value function to $\{c,g,f,X_{0},Y,T\}$.

We now prove if the Inequalities (19) and (20) hold then $J$ is a super-value function to $\{c,g,f,X_{0},Y,T\}$. Multiplying both sides of the inequalities (19) and (20) by $-1$ we get $\forall x\in X_{c},u\in Y,t\in[0,T]$

[TABLE]

Using the previous part of the proof we deduce $-J$ is a sub solution to $\{-c,-g,f,X_{0},Y,T\}$. Thus for any $t\in[0,T]$ and $x\in FR_{f}(X_{0},Y,\{t\})$

[TABLE]

By multiplying both sides of the above inequality by $-1$ we deduce for any $t\in[0,T]$ and $x\in FR_{f}(X_{0},Y,\{t\})$

[TABLE]

Therefore it follows by (23) that $J$ is a super-value function for $\{c,g,f,X_{0},Y,T\}$.∎

Next we give sufficient conditions for the existence of polynomial functions that satisfy Inequalities (17), (18), (19) and (20). This proves the existence of polynomial sub-value and super-value functions but does not show that such functions can arbitrarily well approximate the true value function.

Lemma 4

For $T>0$; a compact set $Y\subset\mathbb{R}^{m}$; a compact set $X_{0}\subseteq\mathbb{R}^{n}$; a polynomial function $g:\mathbb{R}^{n}\to\mathbb{R}$; a function $c\in C^{1}(\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R})$; and $f\in\mathcal{F}_{Y}$; suppose the set $FR_{f}(X_{0},Y,[0,T])$ is bounded. Then there exists a polynomial sub-value function and polynomial super-value function to the optimal control problem $\{c,g,f,X_{0},Y,T\}$.

Proof:

As $FR_{f}(X_{0},Y,[0,T])$ is bounded it follows there exists $R>0$ such that $FR_{f}(X_{0},Y,[0,T])\subset B_{R}$. Now consider the polynomial function

[TABLE]

where $\underline{\alpha}:=\inf_{x\in B_{R},u\in Y,t\in[0,T]}\{\nabla g(x)f(x,u)+c(x,u,t)\}$; which is well defined as the infimum of a differentiable function over a compact set is finite.

To prove the existence of a polynomial sub-value function we show $J_{1}$ satisfies Inequalities (17) and (18), and thus by Proposition 2 we deduce $J_{1}$ is a sub-value function for $\{c,g,f,X_{0},Y,T\}$. Trivially (18) holds. Now for $x\in FR_{f}(X_{0},Y,[0,T])$ and $t\in[0,T]$

[TABLE]

Therefore we conclude $J_{1}$ satisfies (17) and thus is a sub-value function to $\{c,g,f,X_{0},Y,T\}$.

The existence of a super-value function follows by a similar argument and consideration of the function

[TABLE]

where $\underline{\alpha}:=\sup_{x\in B_{R},u\in Y,t\in[0,T]}\{\nabla g(x)f(x,u)+c(x,u,t)\}$. ∎

VII Using SOS To Construct Sub-Value And Super-Value Functions

For an optimal control problem $\{c,g,f,X_{0},Y,T\}$ we would like to find the associated polynomial sub-value and super-value functions with minimum distance under some function metric; and hence are “close” to a true value function. If we choose our function metric as the $L_{1}$ norm we seek to solve the optimization problem:

[TABLE]

where $V^{*}$ is a value function of $\{c,g,f,X_{0},Y,T\}$. To enforce the constraints of the above optimization problem we use Proposition 2; where it was shown if $V_{l}$ satisfies (18) (17) and $V_{u}$ satisfies (20) (19) then $V_{l}$ and $V_{u}$ are sub-value and super-value functions for $\{c,g,f,X_{0},Y,T\}$ respectively. We then are able to tighten the optimization problem to an SOS optimization problem, indexed by $S(T,c,g,f,h_{X},h_{Y},d,\Omega)$:

[TABLE]

where

[TABLE]

Corollary 1

Suppose $V_{u}$ and $V_{l}$ solve $S(T,c,g,f,h_{X},h_{Y},\Omega)$, given in (24). Then $V_{u}$ and $V_{l}$ are super-value and sub-value functions to the optimal control problem $\{c,g,f,X_{0},Y,T\}$ respectively; where $Y\subset\mathbb{R}^{m}$ and $X_{0}\subset\mathbb{R}^{n}$ are such that $FR_{f}(X_{0},Y,[0,T])\subseteq\{x\in\mathbb{R}^{n}:h_{X}(x)\geq 0\}$ and $Y\subseteq\{u\in\mathbb{R}^{m}:h_{Y}(u)\geq 0\}$.

Moreover if $\Omega\subseteq X_{0}$ the following holds,

[TABLE]

where $\varepsilon=\int_{\Omega}V_{u}(x,0)-V_{l}(x,0)dx$ and $V^{*}$ is the value function of the optimal control problem $\{c,g,f,X_{0},Y,T\}$.

Proof:

We first prove $V_{l}$ is a sub-value function by showing $V_{l}$ satisfies the dissipation inequalities (17) and (18); as it follows by Proposition 2 that such a function must be a sub-value function of $\{c,g,f,X_{0},Y,T\}$.

As $k_{0,l}\in\sum_{SOS}$ it follows $k_{0,l}(x)\geq 0$ for all $x\in\mathbb{R}^{n}$. Moreover since a positive function multiplied by a positive function is a postive function we furthermore deduce

[TABLE]

As $FR_{f}(X_{0},Y,[0,T])\subseteq\{x\in\mathbb{R}^{n}:h_{X}(x)\geq 0\}$ the above inequality also holds for all $x\in FR_{f}(X_{0},Y,[0,T])$. Therefore $V_{l}$ satisfies Inequality (18).

As $k_{1,l}\in\sum_{SOS}$ it follows for all $x\in\{y\in\mathbb{R}^{n}:h_{X}(y)\geq 0\}$, $u\in\{w\in\mathbb{R}^{m}:h_{Y}(w)\geq 0\}$, and $t\in\{s\in\mathbb{R}:[s][T-s]\geq 0\}$

[TABLE]

As $FR_{f}(X_{0},Y,[0,T])\subseteq\{x\in\mathbb{R}^{n}:h_{X}(x)\geq 0\}$, $Y\subseteq\{w\in\mathbb{R}^{m}:h_{Y}(w)\geq 0\}$ and $[0,T]=\{s\in\mathbb{R}:[s][T-s]\geq 0\}$ it follows $V_{l}$ satisfies Inequality (17). Therefore we conclude $V_{l}$ is a sub-value function as it satisfies the Inequality (17) and (18). Moreover, it follows by a similar argument to the above that $V_{u}$ is a super-value function.

Finally the error bounds in (25) immediately follows using $V_{l}(x,0)\leq V^{*}(x,0)\leq V_{u}(x,0)$ for all $x\in\Omega\subseteq X_{0}$ and $t\in[0,T]$. ∎

In Lemma 3 we saw how sub-value and super-value functions over- and inner-bound reachable sets. In the next corollary we will show how solutions to the SOS Optimization Problem (24) also over and inner bound reachable sets.

Corollary 2

Suppose $V_{u}$ and $V_{l}$ solve $S(T,0,g,f,h_{X},h_{Y},\Omega)$, given in (24). Let $Y=\{u\in\mathbb{R}^{m}:h_{Y}(u)\geq 0\}$ and $X_{0}=\{x\in\mathbb{R}^{n}:g(x)\leq 1\}$. Suppose for some $X\subset\mathbb{R}^{n}$ such that $BR_{f}(X_{0},Y,\{T\})\subseteq X$ the following holds $FR_{f}(X,Y,[0,T])\subseteq\{x\in\mathbb{R}^{n}:h_{X}(x)\geq 0\}$. Then

[TABLE]

Proof:

By Corollary 1 the functions $V_{u}$ and $V_{l}$ are super-value and sub-value functions to the optimal control problem $\{0,g,f,X,Y,T\}$ where $BR_{f}(X_{0},Y,\{T\})\subseteq X$. Therefore by Lemma 3 the set containments (26) hold. ∎

For reachable set analysis using $S(T,0,g,f,h_{X},h_{Y},d,\Omega)$, given in (24), typically we select $h_{X}=R^{2}-x_{1}^{2}-x_{2}^{2}$ for $R>0$ so $\{x\in\mathbb{R}^{n}:h_{X}(x)\geq 0\}=B_{R}$. Then, assuming the set $BR_{f}(X_{0},f,{T})$ is compact, we select $R>0$ sufficiently large enough for there to exist a compact set $X\subset\mathbb{R}^{n}$ such that $BR_{f}(X_{0},f,{T})\subseteq X$ and $FR_{f}(X,Y,[0,T])\subseteq B_{R}$. Knowledge of the set $X\subset\mathbb{R}^{n}$ is not necessary to construct an outer approximation of the backward reachable set; as by Corollary 2 we have $BR_{f}(X_{0},Y,\{T\})\subseteq\{x\in X:V_{l}(x,0)\leq 1\}\subseteq\{x\in\mathbb{R}^{n}:V_{l}(x,0)\leq 1\}$, where $(V_{u},V_{l})$ solve $S(T,0,g,f,h_{X},h_{Y},d,\Omega)$.

VII-A Numerical Example: Using SOS To Numerically Approximating A Non-Differentiable Value Function

Let $X_{0}=[-8,8]$; $T>0$; $Y=[-1,1]$; $c(x,t)=0$ for all $x\in\mathbb{R}$ and $t>0$; $g(x)=x$; $f(x,u)=xu$ and consider the optimal control problem $\{c,g,f,X_{0},Y,T\}$. It was shown in [17] that the value function of $\{c,g,f,X_{0},Y,T\}$ can be analytically found as

[TABLE]

We note that $V$ is not differentiable at $x=0$ but can be shown to satisfy the associated HJB PDE away from $x=0$. This problem shows how the value function can be non-smooth even for simple optimal control problems with polynomial vector field and cost. We next attempt to find a polynomial, and thus smooth, super-value and sub-value functions of this optimal control problem that is close to the non-smooth value function given in (27) under the $L_{1}$ norm.

We numerically solved the SOS optimization problem $S(T,c,g,f,h_{X},h_{Y},d,\Omega)$ with $T=1$; $c$, $g$ and $f$ the same as the above optimal control problem; $h_{X}=8^{2}-x^{2}$; $h_{Y}(u)=(-1-u)(u-1)$; $d=4$; $\Omega=[-2,2]$. The result is displayed in Figure 1 where the exact value function, given in (27), is plotted as the dotted line and super-value and sub-value functions are plotted as the blue and red line respectively. We see even though the exact value function is discontinuous at $x=0$ the smooth polynomial sub-value is a reasonable tight approximation.

VII-B Numerical Examples: Using SOS To Solve The HJB PDE For Reachable Set Approximation

Example 1

Let us now consider the Van der Pol oscillator defined by the nonlinear ODE:

[TABLE]

To find the forward reachable set for the Van der Pol oscillator we solved the optimization problem $S(T,c,g,f,h_{X},h_{Y},d,\Omega)$, found in (24), with $T=1$; $c=0$; $g(x)=x_{1}^{2}+x_{2}^{2}$; $f(x)=-[x_{2},-x_{1}+x_{2}(1-x_{1}^{2})]^{T}$; $h_{X}(x)=10^{2}-x_{1}^{2}-x_{2}^{2}$; $h_{Y}(u)=0$; $d=4$ and $\Omega=[-2,2]\times[-2,2]$. The sublevel sets $\{x\in\mathbb{R}^{n}:V_{u}(x,0)\leq 1\}$ and $\{x\in\mathbb{R}^{n}:V_{l}(x,0)\leq 1\}$, where $(V_{u},V_{l})$ solve the above optimization problem, are then plotted in Figure 2 as the black line and green line respectively. As shown in Corollary 2 these sublevel sets are over and under set approximations of $BR_{f}(X_{0},Y,\{T\})$, which was shown to be equal to $FR_{-f}(X_{0},Y,\{T\})$ in Lemma 2, where $X_{0}=\{x\in\mathbb{R}^{n}:g(x)\leq 1\}$. This is clearly demonstrated in Figure 2 where the red points represent initial points contained inside the set $X_{0}$ and blue points represent points the solution map can transition to at time $T=1$ starting in $X_{0}$; where both sets of points were approximately found from forward time integrating (28).

Example 2

Let us consider the linear ODE:

[TABLE]

where $A=\begin{bmatrix}0&-1\\ 1&0\end{bmatrix}$. Since the eigenvalues of $A$ are $\pm i$ it follows (29) produces non-stable circular trajectories for fixed input $\mathbf{u}(t)\equiv u\in\mathbb{R}^{m}$.

To find the forward reachable set for this linear ODE (29) for fixed input $\mathbf{u}(t)\equiv 1$ we solved the optimization problem $S(T,c,g,f,h_{X},h_{Y},d,\Omega)$, found in (24), for both $d=4$ and $d=4$ with $T=5$; $c=0$; $g(x)=(x_{1}-1.5)^{2}+x_{2}^{2}$; $f(x)=-Ax$; $h_{X}(x)=10^{2}-x_{1}^{2}-x_{2}^{2}$; $h_{Y}(u)=0$; and $\Omega=[-3,3]\times[-3,3]$. We plotted the 1-sublevel sets at time [math] of the solutions to these optimization problem, $V_{u}$ and $V_{l}$, in Figure 3 as the black line and green line respectively; where the dotted lines are for $d=3$ and filled lines for $d=4$. Here the red points represent initial points contained inside the set $X_{0}=\{x\in\mathbb{R}^{n}:g(x)\leq 1\}$ and blue points represent points the solution map can transition to at time $T=5$ starting in $X_{0}$; where both sets of points were approximately found from forward time integrating (29). As expected, by Corollary 2, we see these sublevel sets under and over approximate the reachable set respectively. We also see increasing the degree makes our approximations tighter.

We have furthermore approximated the forward reachable set of the linear ODE (29) when the input is allowed to vary but constrained inside the set $Y=[-2,2]$. To do this we solved the optimization problem $S(T,c,g,f,h_{X},h_{Y},d,\Omega)$, found in (24), with $T=0.5$; $c=0$; $g(x)=(x_{1}-1.5)^{2}+x_{2}^{2}$; $f(x)=-Ax$; $h_{X}(x)=4^{2}-x_{1}^{2}-x_{2}^{2}$; $h_{Y}(u)=(u+2)(2-u)$; $d=2$ and $\Omega=[-3,3]\times[-3,3]$. In Figure 4 we then plotted $\{x\in\mathbb{R}^{2}:V_{l}(x,0)\leq 1\}$ as the green line, where $(V_{u},V_{l})$ solves the above optimization problem. By Corollary 2 the set $\{x\in\mathbb{R}^{2}:V_{l}(x,0)\leq 1\}$ over approximates the set $BR_{f}(X_{0},Y,\{T\})$, shown in Lemma 2 to be equal to $FR_{-f}(X_{0},Y,\{T\})$, where $X_{0}=\{x\in\mathbb{R}^{n}:g(x)\leq 1\}$. This is demonstrated in Figure 4 as the terminal points of the solution map at time $T=0.5$, represented by the blue points, are all contained inside the green line.

VIII Conclusion

In this paper we have shown if a function satisfies dissipation inequalities then it is a sub-value or super-value function to an optimal control problem. Further to this we have given sufficient conditions for the existence of polynomial sub-value and super-value functions to optimal control problems. An SOS optimization problem was proposed that is solved by sub-value and super-value functions of an optimal control problem that have minimum $L_{1}$ norm. It was shown how this SOS optimization problem is able to construct outer and inner set approximations of reachable sets.

Acknowledgements

This work was supported by the National Science Foundation under grants No. 1538374 and 1739990.