Convexity and monotonicity in nonlinear optimal control under uncertainty

TL;DR

This paper extends convex optimal control methods to certain nonlinear systems with uncertainty, showing that under specific conditions, the problem remains convex and solvable with standard optimization tools.

Contribution

It generalizes convex control frameworks to nonlinear dynamics with monotonicity and purification conditions, enabling global solutions for a broader class of systems.

Findings

Convex reformulation applies to nonlinear systems with specific monotonicity.

Finite-dimensional subproblems in control methods remain convex.

Numerical example demonstrates practical applicability.

Abstract

We consider the problem of finite-horizon optimal control design under uncertainty for imperfectly observed discrete-time systems with convex costs and constraints. It is known that this problem can be cast as an infinite-dimensional convex program when the dynamics and measurements are linear, uncertainty is additive, and the risks associated with constraint violations and excessive costs are measured in expectation or in the worst case. In this paper, we extend this result to systems with convex or concave dynamics, nonlinear measurements, more general uncertainty structures and other coherent risk measures. In this setting, the optimal control problem can be cast as an infinite-dimensional convex program if (1) the costs, constraints and dynamics satisfy certain monotonicity properties, and (2) the measured outputs can be reversibly `purified' of the influence of the control inputs through Q- or Youla-parameterization. The practical value of this result is that the finite-dimensional subproblems arising in a variety of suboptimal control methods, notably including model predictive control and the Q-design procedure, are also convex for this class of nonlinear systems. Subproblems can therefore be solved to global optimality using convenient modeling software and efficient, reliable solvers. We illustrate these ideas in a numerical example.

Figures

Tables

Table 1. TABLE I: Example risk measures for a random variable X 𝑋 X .

Risk measure	Definition of $ℛX$	Risk acceptable if…
Expected value	$𝐄X$	$X\le 0$ on average
Predicted value	prediction $\hat{X}$ of $X$	$\hat{X}\le 0$
Worst-case value	$supX$	$X\le 0$ almost surely
Value-at-risk, ${\text{VaR}}_{\beta }\left(X\right)$	$inf\{\alpha \mid$ $𝐏\{X\le \alpha \}\ge \beta \}$	$𝐏\left\{X\le 0\right\}\ge \beta$
Conditional    value-at-risk	${inf}_{\alpha }\{\alpha +$ $𝐄{(X-\alpha )}_{+}/(1-\beta )\}$	$𝐄[X\mid X\ge$ ${\text{VaR}}_{\beta }\left(X\right)]\le 0$

Table 2. TABLE II: Approximations of the risk measures in Table I based on samples X ( 1 ) , … , X ( N ) superscript 𝑋 1 … superscript 𝑋 𝑁 X^{(1)},\dots,X^{(N)} from the distribution of X 𝑋 X .

Risk measure, $ℛX$	Sample-based approximation, ${\widehat{ℛ}}^N(X^{\left(1\right)},\dots ,X^{\left(N\right)})$
Expected value	$\left(X^{\left(1\right)}+\cdots +X^{\left(N\right)}\right)/N$
Predicted value	prediction based on $X^{\left(1\right)},\dots ,X^{\left(N\right)}$
Worst-case value	$\max \{X^{\left(1\right)},\dots ,X^{\left(N\right)}\}$
Value-at-risk	$\max \{X^{\left(1\right)},\dots ,X^{\left(N\right)}\}$
Conditional    value-at-risk	${inf}_{\alpha }\{\alpha +({(X^{\left(1\right)}-\alpha )}_{+}+\dots$ $+{(X^{\left(N\right)}-\alpha )}_{+})/\left(N(1-\beta )\right)\}$

Table 3. TABLE III: Sample frequency of x t ≤ 1 subscript 𝑥 𝑡 1 x_{t}\leq 1

Controller	Frequency
Optimal affine	99.5%
Open-loop    scenario MPC	99.3%
Nonlinear $Q$-design	98.9%
Closed-loop    scenario MPC	98.7%
Certainty-equivalent    MPC	74.6%

Equations

$$x_0$$

$$x_t$$

$$y_0$$

$$y_t$$

$$u_t={\pi }_t(y_0,\dots ,y_t)$$

$$x=\left[\begin{array}{c}{x}_0\\ x_1\\ ⋮\\ x_T\end{array}\right]=\left[\begin{array}{c}{\varphi }_0({\delta }_0)\\ {\varphi }_1(u_0,{\delta }_{0:1})\\ ⋮\\ {\varphi }_T(u_{0:T-1},{\delta }_{0:T})\end{array}\right]=\varphi (u,\delta ).$$

$${\varphi }_0({\delta }_0)$$

$${\varphi }_1(u_0,{\delta }_{0:1})$$

$${\varphi }_t(u_{0:t-1},{\delta }_{0:t})$$

$$t=2,\dots ,T.$$

$$y=\left[\begin{array}{c}{y}_0\\ y_1\\ ⋮\\ y_{T-1}\end{array}\right]=\left[\begin{array}{c}{\psi }_0({\delta }_0)\\ {\psi }_1(u_0,{\delta }_{0:1})\\ ⋮\\ {\psi }_{T-1}(u_{0:T-2},{\delta }_{0:T-1})\end{array}\right]=\psi (u,\delta ),$$

$${\psi }_0({\delta }_0)$$

$${\psi }_t(u_{0:t-1},{\delta }_{0:t})$$

$$t=1,\dots ,T-1.$$

$$u=\pi (y)=\left[\begin{array}{c}{\pi }_0(y_0)\\ {\pi }_1(y_{0:1})\\ ⋮\\ {\pi }_{T-1}(y_{0:T-1})\end{array}\right].$$

$$c_0(x,u,\delta )$$

$$c_j(x,u,\delta )\le 0,j=1,\dots ,J.$$

$${\mathcal{R}}_j{c}_j(x,u,\delta )\le 0.$$

$$P\left\{c_j(x,u,\delta )\le 0\right\}\ge \beta .$$

$$E(c_j(x,u,\delta )-\alpha {)}_{+}\le -\alpha (1-\beta ),$$

$$\begin{array}[]{ll}\underset{\pi\in\Pi}{\operatorname*{minimize}}&\mathcal{R}_{0}c_{0}(x,u,\delta)\\ \mbox{subject to}&\mathcal{R}_{j}c_{j}(x,u,\delta)\leq 0,\quad j=1,\dots,J\\ &x=\phi(u,\delta)\\ &u=\pi(y)\\ &y=\psi(u,\delta).\end{array}$$

$$\begin{array}[]{ll}\underset{\tilde{u}}{\operatorname*{minimize}}&\mathcal{R}_{0}c_{0}(\phi(\tilde{u},\delta),\tilde{u},\delta)\\ \mbox{subject to}&\mathcal{R}_{j}c_{j}(\phi(\tilde{u},\delta),\tilde{u},\delta)\leq 0,\quad j=1,\dots,J.\end{array}$$

$$u\mapsto c_j(\varphi (u,\delta ),u,\delta ),j=0,\dots ,J$$

$$x_t=\left[\begin{array}{c}{x}_t^{\mathrm{aff}}\\ x_t^{\mathrm{cvx}}\end{array}\right]$$

$$(x^{\mathrm{aff}},x^{\mathrm{cvx}},u)\mapsto c_j((x^{\mathrm{aff}},x^{\mathrm{cvx}}),u,\delta )$$

$$z_t^{\text{cvx}}$$

$$z_t^{\text{ccv}}$$

$$x_t^{\mathrm{cvx}}=\left[\begin{array}{c}{z}_t^{\text{cvx}}\\ -z_t^{\text{ccv}}\end{array}\right]$$

$$=h_t(x_{t-1}^{\mathrm{aff}},x_{t-1}^{\mathrm{cvx}},u_{t-1},{\delta }_t),$$

$$p_t({\psi }_{0:t}(u_{0:t-1},{\delta }_{0:t}),u_{0:t-1})={\xi }_t({\delta }_{0:t})$$

$$⟺$$

$$y_0=g_0(x_0,{\delta }_0)=g_0(f_0({\delta }_0),{\delta }_0).$$

$$p_0(y_0)=y_0,q_0(e_0)=e_0,{\xi }_0({\delta }_0)=g_0(f_0({\delta }_0),{\delta }_0).$$

$${\pi }_0(y_0)$$

$${\pi }_t(y_{0:t})$$

$$t=1,\dots ,T-1.$$

Full text

I Introduction

We consider the problem of operating a system that evolves in discrete time steps over a finite horizon under uncertainty. Our goal is to design an output feedback control policy that minimizes a convex cost while satisfying convex constraints. This problem has been studied in the stochastic setting, where uncertainty is modeled probabilistically, the expected cost is minimized, and constraints are enforced in expectation or in probability. [1, 2, 3, 4, 5, 6] It has also been studied in the robust setting, where uncertainty is assumed to come from a given set, the worst-case cost is minimized, and constraints are enforced for all possible realizations of the uncertain influences on the system. [7, 8, 9, 10, 11] We accommodate both approaches in this paper, as well as others, adopting the flexible view of risk developed in [12, 13, 14, 15, 16, 17].

Optimal control under uncertainty is a hard problem, even when the costs and constraints are convex. This is due in part to the fact that the optimization variable is a policy, a collection of functions that map measured system outputs into control inputs at each time step. In the stochastic setting, optimal policies can, in principle, be found analytically using stochastic dynamic programming. [18] In practice, analytical solution is typically limited to systems of very low dimension. Notable exceptions are linear systems with additive uncertainty, no constraints, and costs that are either quadratic [19, 20] or exponential-of-quadratic [21, 22].

When analytical solution is impractical, various methods can generate suboptimal solutions that often perform well. Some examples are classical linear feedback control design, reinforcement learning [23, 24, 25, 26] and approximate dynamic programming [4, 27, 28, 29], approximation methods for multistage stochastic programming [6, 30, 31, 32, 33] and robust optimization with recourse [33, 34, 35, 36], the $Q$-design procedure [37, 38, 39, 40, 41, 42], and model predictive control (MPC) in its certainty-equivalent [43, 44, 45], stochastic [46, 47, 48, 49, 50, 51, 52], and robust [53, 54, 55, 56, 57] variants. When perfect state information is not available, control methods may be paired with a state estimator such as a linear [58, 59], extended [60, 61, 62] or unscented [63, 64] Kalman filter or a particle filter [65, 66].

These methods vary widely in their scope, scalability and performance. A common theme, however, is that they tend to work best for linear systems. This is due in part to the fact that for linear systems, (an equivalent transformation of) the optimal control problem is convex. [41] Suboptimal control methods often involve numerically solving optimization subproblems generated by the original optimal control problem. The convexity of the original problem typically carries over to the subproblems, allowing them to be efficiently and reliably solved to global optimality. When subproblems are nonconvex, however, guarantees of global optimality are generally unavailable, and solvers and initial guesses may need to be carefully tailored to the applications at hand.

In [41], Skaf and Boyd demonstrate that for linear systems with additive uncertainty, the control design problem can be transformed to an infinite-dimensional convex program. Their method hinges on a change of variables related to the $Q$- or Youla-parameterization [67, 68, 69] and to purified output feedback control [2, 10, 34]. When perfect state information is available, this change of variables parameterizes state feedback policies by equivalent disturbance feedback policies. Similar arguments have justified the use of (typically affine) disturbance feedback policies in robust and stochastic MPC of perfectly-observed linear systems. [42, 47, 70, 71, 72, 73, 74]

For nonlinear systems, the optimal control problem is widely understood to be nonconvex due to nonlinear equality constraints introduced by nonlinear dynamics. Equality constraints can be eliminated, however, by iteratively applying the dynamics to express the state trajectory in terms of the control and exogenous input trajectories. In [75], Rantzer and Bernhardsson observe that in convex-monotone systems, where the dynamics are convex and nondecreasing in the state and control input, the state trajectory is convex in the control input trajectory. In [76], Schmitt et al. generalize this result to convex-state-monotone systems, where the dynamics need not be monotone in the control input.

An immediate consequence of the observations in [75, 76] is that for some nonlinear systems, open-loop optimal control (where the decision variable is a fixed control trajectory, rather than a feedback policy) is a convex optimization problem. This holds for convex-state-monotone systems in particular, provided the cost and constraints are nondecreasing in the states. This raises two further questions: Are there other nonlinear systems for which open-loop optimal control is convex? What about closed-loop optimal control, where we optimize over policies?

This discussion motivates the definition of a convex system as one for which open-loop optimal control is a convex optimization problem. After setting the stage in §II, we establish three results for convex systems in this paper.

Characterization (§III): Systems with mixed convex and linear dynamics are convex systems, provided (a) any linear dynamics are independent of states with nonlinear dynamics, and (b) the cost, constraints and nonlinear dynamics are nondecreasing in the states with nonlinear dynamics. Concave dynamics can also be accommodated. 2. 2.

Convex closed-loop design (§IV): If the measured outputs can be reversibly ‘purified’ of the influence of the control inputs, then the closed-loop optimal control problem for convex systems can be transformed to an equivalent infinite-dimensional convex program. The transformation involves changing variables to policies in the purified outputs via $Q$- or Youla-parameterization. 3. 3.

Approximate solution (§V–VI): The finite-dimensional subproblems arising in a variety of suboptimal control methods, notably including MPC and the $Q$-design procedure, are convex for convex systems. Subproblems can therefore be solved to global optimality using convenient modeling software and efficient, reliable solvers. We illustrate this in a numerical example.

II Problem statement

II-A System

We consider a system to be operated over a finite discrete time span $t=0,\dots,T$. We can influence the system through the control inputs $u_{0},\dots,u_{T-1}$. The system is also influenced by exogenous inputs $\delta_{0},\dots,\delta_{T}$, which are generally uncertain.111We view the exogenous input $\delta_{t}$ as a random vector defined on an underlying probability space. We do not assume that the joint distribution of the exogenous input trajectory is known, but we do assume this distribution is independent of the control input trajectory. The exogenous inputs may include process disturbances, sensor noise, initial states, uncertain model parameters, prices, command or reference signals, etc. The exogenous inputs could come from bounded or unbounded sets, and need not be independent or identically distributed over time.

The control and exogenous inputs determine the states $x_{0},\dots,x_{T}$ through the system dynamics:

[TABLE]

We observe the system through the measured outputs

[TABLE]

We assume that the dynamics mappings $f_{0},\dots,f_{T}$ and measurement mappings $g_{0},\dots,g_{T-1}$ are known. At each time $t$, the controller receives the measured output $y_{t}$. It decides the control input

[TABLE]

by evaluating an output feedback control law $\pi_{t}$. The control policy $\pi=(\pi_{0},\dots,\pi_{T-1})$ is designed in advance; this design problem is the subject of this paper.

To simplify specification of the policy design problem, we work with the input, state and output trajectories. By iteratively applying the dynamics, the state trajectory can be expressed in terms of the control and exogenous input trajectories as

[TABLE]

Here the subscript $t_{1}:t_{2}$ denotes a trajectory from time $t_{1}$ to $t_{2}$. For example, $\delta_{0:T}=(\delta_{0},\dots,\delta_{T})=\delta$ and $u_{0:T-1}=(u_{0},\dots,u_{T-1})=u$. The input-state mappings $\phi_{t}$ are defined by the recursion

[TABLE]

Similarly, the measured output trajectory can be written as

[TABLE]

where the input-output mappings $\psi_{t}$ are defined recursively by

[TABLE]

We also write the control input trajectory as

[TABLE]

Figure 1 illustrates the system.

II-B Cost, constraints and risk measures

We are interested in designing the output feedback control policy $\pi=(\pi_{0},\dots,\pi_{T-1})$. The policy must be causal, meaning each control law can depend on outputs from the past or present, but not the future. We denote the set of causal output feedback policies by $\Pi$. Our goal is to find a $\pi\in\Pi$ that minimizes a cost

[TABLE]

while satisfying constraints

[TABLE]

We assume that each scalar-valued function $c_{0},\dots,c_{J}$ is convex in $(x,u)$ for all $\delta$.

Because the exogenous inputs are uncertain, the goals of minimizing cost and satisfying constraints are ambiguous. Should they be accomplished on average with respect to the distribution of $\delta$, for all possible realizations of $\delta$, or somewhere in between? Resolving these ambiguities amounts to choosing measures of risk.222For more on measuring risk, a rich subject that has received much recent attention, we refer the reader to [12, 13, 14, 15, 16, 17]. A risk measure $\mathcal{R}$ is a functional that maps uncertain scalars into deterministic scalars (or possibly $\infty$). It quantifies the risk associated with positive realizations of its uncertain argument. In the language of [12, 13], the risk measure $\mathcal{R}_{j}$ should be associated with constraint $j$ if the designer views the risk of constraint violation as acceptable whenever

[TABLE]

The risks associated with excessive costs can be treated similarly to the risks associated with constraint violations, because minimizing $c_{0}(x,u,\delta)$ is equivalent to minimizing a deterministic scalar $\tau$ subject to the constraint $c_{0}(x,u,\delta)-\tau\leq 0$.

Table I describes five risk measures commonly used in optimization under uncertainty. The expected value is risk-neutral: it weighs constraint violations and slacks equally, modeling the risk of constraint violation as acceptable if slacks balance out violations on average. The worst-case value, by contrast, is maximally risk-averse: it models any possibility of constraint violation as unacceptable. The value-at-risk at confidence level $\beta\in[0,1]$ ($\text{VaR}_{\beta}$) is the smallest threshold that, with 100$\beta$% confidence, will not be exceeded. Typical values of $\beta$ are 0.9, 0.95 and 0.99. $\text{VaR}_{\beta}$ generates chance constraints of the form

[TABLE]

The conditional value-at-risk at confidence level $\beta$ (${\rm CVaR}_{\beta}$) is the conditional expectation, conditioned on the event that $\text{VaR}_{\beta}$ is exceeded. [14, 15] ${\rm CVaR}_{\beta}$ always upper bounds $\text{VaR}_{\beta}$. The ${\rm CVaR}_{\beta}$ of $c_{j}(x,u,\delta)$ can be constrained by minimizing a scalar $\alpha$ subject to

[TABLE]

where $(\cdot)_{+}$ denotes the positive part, $\max\left\{0,\cdot\right\}$. An optimal $\alpha^{\star}$ upper bounds $\text{VaR}_{\beta}c_{j}(x,u,\delta)$; this bound is usually tight. [15]

In this paper, we leave the choice of risk measures to the designer. This flexibility makes the results in the following sections applicable in a range of settings, including the robust and stochastic frameworks and others. We do, however, impose one restriction on the risk measures: we assume that they are coherent in the sense of [12, 13, 16, 17]. By this, we mean that each risk measure is convex, nondecreasing, lower semicontinuous, and preserves certainty (i.e., $\mathcal{R}(C)=C$ for constant $C$). We make this assumption primarily because coherent risk measures preserve convexity: if $\mathcal{R}_{j}$ is coherent and $(x,u)\mapsto c_{j}(x,u,\delta)$ is convex for all $\delta$, then $(x,u)\mapsto\mathcal{R}_{j}c_{j}(x,u,\delta)$ is convex. [16]

All but one of the risk measures mentioned above are coherent. The exception is $\text{VaR}_{\beta}$, which is convex only in structured special cases. We note, however, that $\text{VaR}_{\beta}$ can be approximated in a convexity-preserving fashion using sampling [77, 78, 79, 80, 81, 82] or conservative upper bounds such as ${\rm CVaR}_{\beta}$ [6, 50].

In summary, we are interested in the following problem of optimal control under uncertainty:

[TABLE]

This problem is intractable in its current form. This is due in part to the fact that the decision variable $\pi$ is infinite-dimensional, an obstacle that we will address approximately in §V. Problem (OC) is also complicated by the interdependence between the state, output and control trajectories. We will unravel this interdependence in §IV using a nonlinear change of variables. First, we will consider the case of open-loop control, where the control laws $\pi_{t}$ are restricted to be constant.

III Convex systems

To understand when closed-loop control design is a convex optimization problem, we begin by building intuition in the simpler context of open-loop control. In this context, a (finite-dimensional) vector of control inputs $\tilde{u}=(\tilde{u}_{0},\dots,\tilde{u}_{T-1})$ is decided in advance and implemented without feedback. This amounts to drastically restricting the search space from the set of all causal output feedback policies to the subspace of constant policies. With this restriction, the open-loop optimal control problem is to

[TABLE]

This is a finite-dimensional optimization problem in the vector $\tilde{u}$. Because coherent risk measures preserve convexity [16], Problem (OLOC) is convex if each cost and constraint function $c_{j}$, when composed with the input-state mapping $\phi$, is convex in $u$ for all $\delta$. This motivates the following definition.

Definition 1 (Convex system).

We call the system with dynamics (1), cost (3) and constraints (4) a convex system if the functions

[TABLE]

are convex for all $\delta$.

By definition, open-loop optimal control of convex systems is a convex optimization problem. In §IV, we will show that closed-loop optimal control of convex systems is also convex, provided the outputs can be reversibly ‘purified’ of the influence of the control inputs. First, we will characterize a class of convex systems.

III-A Characterizing convex systems

To understand when a system is convex, we need to understand how the states and controls propagate through the dynamics (1) into the cost (3) and constraints (4). If the dynamics are linear, then this process is straightforward. The state trajectory $x$ is affine in the control trajectory $u$ for all $\delta$. Convexity is preserved under composition with an affine mapping [83], so for linear systems $u\mapsto c_{j}(\phi(u,\delta),u,\delta)$ is convex for all $\delta$.

When the system is nonlinear, the process is less straightforward. We do not explore it in general here. Instead, we consider a class of systems with the following dynamics:

[TABLE]

For this class of systems, convexity can be established using simple composition rules. An important restriction is that the states $x^{\text{\rm aff}}_{t}$ with linear dynamics are independent of the nonlinear states $x^{\rm cvx}_{t}$. This ensures that the trajectory $x^{\text{\rm aff}}=(x^{\text{\rm aff}}_{0},\dots,x^{\text{\rm aff}}_{T})$ is affine in the control trajectory. Any states with linear dynamics that depend on nonlinear states are included in $x^{\rm cvx}_{t}$.

Theorem 1.

The system with dynamics (5), cost (3) and constraints (4) is a convex system if the following conditions hold for all $\delta$.

For $t=1,\dots,T$, each row of the nonlinear dynamics mapping $h_{t}$ is

1. (a)

jointly convex in $(x^{\rm aff}_{t-1},x^{\rm cvx}_{t-1},u_{t-1})$, and 2. (b)

nondecreasing in each element of $x^{\rm cvx}_{t-1}$. 2. 2.

For $j=0,\dots,J$, the function

[TABLE]

is nondecreasing in each element of $x^{\rm cvx}=(x^{\text{\rm cvx}}_{0},\dots,x^{\text{\rm cvx}}_{T})$.

Appendix A contains a proof of Theorem 1. The proof hinges on the fact that the composition of a convex nondecreasing function with a convex function is convex. [83]

Three remarks are in order. First, system (5) is a straightforward generalization of the convex-state-monotone system studied in [76], which in turn generalizes the convex-monotone system studied in [75]. A precise name for system (5) would be convex-nonlinear-state-monotone, since the monotonicity requirement applies only to the states with nonlinear dynamics. The purpose of this generalization is to absorb the class of linear systems for which optimal control was shown in [41] to be convex. In particular, the linear-quadratic system is not convex-monotone or convex-state-monotone in general, but is a special case of system (5) with linear dynamics, additive uncertainty, no constraints and quadratic cost.

Second, concave dynamics can be included in system (5) after a sign change. For example, we consider a system with

[TABLE]

where each row of $h^{\text{cvx}}_{t}$ is nondecreasing in $z^{\text{cvx}}_{t-1}$, nonincreasing in $z^{\text{ccv}}_{t-1}$ and convex. Similarly, each row of $h^{\text{ccv}}_{t}$ is nondecreasing in $z^{\text{ccv}}_{t-1}$, nonincreasing in $z^{\text{cvx}}_{t-1}$ and concave. If we define the nonlinear state as

[TABLE]

then each row of $h_{t}$ is nondecreasing in $x^{\rm cvx}_{t-1}$ and convex, as required by Theorem 1.

Third, system (5) includes a rich class of nonlinear systems. Provided careful attention is paid to curvature and monotonicity, it can admit exponentials and logarithms, quadratic forms, roots, powers, nonsmooth functions such as maxima, minima and absolute values, and sums and compositions of the above. Many more examples of nonlinear convex and concave functions can be found in §3 of [83]. Systems with nonlinear convex dynamics have arisen in applications ranging from cancer and HIV treatment scheduling [75] to voltage control in power systems [75] to freeway congestion management [76] to energy storage control [84, 85]. The drug treatment model in [75] is a special case of a more general class of bilinear systems that, through a logarithmic transformation, can be cast as convex systems.

IV Convex closed-loop optimal control

By contrast to the open-loop optimal control problem (OLOC), the equality constraints in the closed-loop optimal control problem (OC) cannot be easily eliminated. This is due to the complicated interdependence between the state, output and control input trajectories. Under some conditions on the uncertainty structure, however, this interdependence can be disentangled through a nonlinear change of variables related to the $Q$- or Youla-parameterization [67, 68, 69] and purified output feedback control [2, 10, 34]. We now establish sufficient conditions for this change of variables to be possible.

IV-A Purifiability and Q-parameterization

Definition 2 (Purifiable).

We call the system with dynamics (1) and measurements (2) purifiable if for each $t=1,\dots,T-1$, there exist mappings $p_{t}$, $q_{t}$ and $\xi_{t}$ such that for all $u_{0:t-1}$ and $\delta_{0:t}$,

[TABLE]

We call $e_{t}=\xi_{t}(\delta_{0:t})$ the purified output and $p_{t}$ the purifier.

We note for future reference that $y_{0}$ is a function of $\delta_{0}$ only:

[TABLE]

Without loss of generality, therefore, we define $p_{0}$, $q_{0}$ and $\xi_{0}$ such that

[TABLE]

Theorem 2.

If the system with dynamics (1) and measurements (2) is purifiable, then there exists a one-to-one correspondence between causal output feedback policies $\pi$ and causal policies $Q$ in the purified output $e=\xi(\delta)$. Furthermore, given a causal $Q$ in $e$, the unique corresponding causal $\pi$ in $y$ can be constructed from the following recursion:

[TABLE]

Theorem 2 is closely related to the nonlinear discrete-time Youla parameterization presented by Wu and Lall in [69]. We include a proof in Appendix B for completeness. We give some examples of purifiable nonlinear systems in §IV-B.

Theorem 2 establishes that if the system is purifiable, then we can optimize over policies $Q$ in the purified output $e=\xi(\delta)$. We interpret $e_{t}$ as what remains of the output $y_{t}$ when the effect of the control inputs $u_{0},\dots,u_{t-1}$ has been removed. Given a causal $Q$, the unique corresponding causal $\pi$ can be recovered. We denote the set of causal $Q$ by $\mathcal{Q}$.

Under this change of variables, an equivalent reformulation of the closed-loop optimal control problem (OC) is to

[TABLE]

The equality constraints can now be eliminated, giving another equivalent problem:

[TABLE]

Problem (OC-$Q$) is structurally identical to the open-loop optimal control problem (OLOC), except that the optimization is over (infinite-dimensional) purified output feedback policies $Q$ rather than (finite-dimensional) control input trajectories $\tilde{u}$. It follows that the two problems are convex under the same conditions, namely for convex systems. This observation, together with Theorem 2, gives the following result.

Corollary 3.

For purifiable systems, the output feedback policy design problem (OC) and the purified output feedback policy design problem (OC-$Q$) are equivalent. For convex systems, Problem (OC-$Q$) is convex.

We will see in §V that Corollary 3 establishes the basic tractability of a variety of suboptimal control schemes for nonlinear convex systems. First, we provide a few examples of purifiable systems.

IV-B Purifiability examples

Purifiability is essentially an invertibility property on the input-output mapping $\psi$ with respect to the exogenous inputs. The requirements for purifiability are (1) at each time step, the influence of the control input history can be removed from the current output, possibly using the output history; and (2) this process must be reversible, in the sense that the current output can be reconstructed from the purified output history and the control input history. To ground this notion, we now provide some concrete examples of purifiable systems. This list is not exhaustive.

Measured exogenous inputs

If the exogenous inputs are measured exactly ($y_{t}=\delta_{t}$), then the system is purifiable with

[TABLE]

Pure estimation

If the controller can observe the system but not influence it, then the states and outputs can be expressed as

[TABLE]

In this case, the system is trivially purifiable with $e_{t}=y_{t}$. This implies that various constrained estimation problems can be put in the form of Problem (OC-$Q$). For example, we consider the problem of designing a state estimator $\pi$ such that $u_{t}=\pi_{t}(y_{0:t})$ minimizes the mean squared error in estimating $x_{t}$, with the prior knowledge that $x_{t}\succeq 0$ almost surely. This can be put in the form of Problem (OC-$Q$) by setting $c_{0}(x,u)=\left\|x-u\right\|_{2}^{2}$, $\mathcal{R}_{0}=\mathop{{\bf E{}}}$, $c_{1}(u)=\max_{i,t}\left\{-(u_{t})_{i}\right\}$ and $\mathcal{R}_{1}=\sup$.

Perfect state information, invertible dynamics

If the states are measured exactly ($y_{t}=x_{t}$) and the dynamics are invertible in the exogenous inputs, i.e., there exist mappings $f_{t}^{-1}$ such that

[TABLE]

then the system is purifiable with

[TABLE]

In the special case of additive disturbances, $x_{t}=f_{t}(x_{t-1},u_{t-1})+\delta_{t}$, the purifier reduces to

[TABLE]

Deterministic dynamics, invertible measurements

If the initial state is measured exactly ($y_{0}=x_{0}$), the dynamics are deterministic,

[TABLE]

and the measurements are invertible in the exogenous inputs, i.e., there exist mappings $g_{t}^{-1}$ such that

[TABLE]

then the system is purifiable with

[TABLE]

In the special case of additive noise, $y_{t}=g_{t}(x_{t},u_{t-1})+\delta_{t}$, the purifier reduces to

[TABLE]

State-affine dynamics and measurements, additive uncertainty

If the dynamics and measurements have the form

[TABLE]

then it can be shown that the input-output mappings are additive:

[TABLE]

In this case, the system is purifiable with

[TABLE]

V Approximate solution methods

Although Problem (OC-$Q$) is convex for convex systems, it remains challenging for two reasons. First, the decision variable $Q$ is infinite-dimensional. Second, the risk measures $\mathcal{R}_{0},\dots,\mathcal{R}_{J}$ may be difficult to compute, or even ill-defined if some distributional information is lacking. We discuss methods for addressing the infinite-dimensionality in §V-A and for approximating risk measures in §V-B. The upshot of this discussion is that several suboptimal control methods that perform well for linear systems can be applied to nonlinear convex systems using finite-dimensional convex optimization.

V-A Finite-dimensional restrictions

V-A1 Open-loop model predictive control

A simple, effective method for overcoming the challenge of infinite dimensionality is open-loop MPC. In this method, at each time step we solve a version of the (finite-dimensional) open-loop optimal control problem (OLOC) over a truncated, receding horizon. This generates a planned control input trajectory. We implement the first step in this plan, the system evolves, we update the state estimate and the process repeats. As discussed in §III, open-loop optimal control is convex for convex systems, even those with nonlinear dynamics.

In one common variant of open-loop MPC, typically called certainty-equivalent MPC, the subproblem at each time step is solved under a single prediction of the disturbance trajectory. [43, 44, 45] This amounts to measuring risk with the predicted value risk measure discussed in §II-B. Other risk measures can be used in open-loop MPC, however, and can significantly improve performance. [46, 48, 49, 50, 51, 52, 53, 54, 55, 57] Risk measures can be approximated if necessary, as discussed in §V-B.

Open-loop MPC has several advantages. First, it often performs well in practice. Second, the design process is straightforward and intuitive; the primary design decisions are the prediction horizon, the terminal cost and/or constraints with which to augment the subproblems, and the algorithms for prediction and state estimation. A third advantage, and perhaps an underappreciated one, is that open-loop MPC admits very general uncertainty structures, including additive, multiplicative and others. In particular, since it does not require the Youla-type change of variables discussed in §IV-A, open-loop MPC can be applied to systems that are not purifiable.

Open-loop MPC also has several disadvantages. First, its implementation is computationally intensive due to the use of online optimization; this can limit its scalability. (We note, however, that for linearly constrained linear systems with quadratic costs, open-loop MPC can be implemented efficiently using custom solvers that exploit the problem structure. [45]) Second, open-loop MPC yields no closed-form expression for the control policy $\pi$; this complicates analysis of closed-loop stability, robustness, etc. Third, the method’s performance can suffer somewhat due to the open-loop structure of the optimal control subproblems. This structure ignores the controller’s opportunity to respond to future information as it becomes available, i.e., the controller’s recourse. The $Q$-design procedure partially addresses these disadvantages.

V-A2 Q-design

In open-loop MPC, the policy design procedure is straightforward (choosing a few parameters and subroutines), but implementation is computationally intensive due to the use of online optimization. In the $Q$-design procedure [37, 38, 39, 40, 41, 42, 73], by contrast, policy design is computationally intensive, but implementation is extremely efficient. In particular, no online optimization is needed. Another important distinction is that unlike MPC, $Q$-design yields a closed-form control policy. This facilitates analysis and simulation of the closed-loop system.

In $Q$-design, we design a suboptimal causal purified output feedback policy

[TABLE]

Here $Q^{(1)},\dots,Q^{(K)}\in\mathcal{Q}$ are causal basis policies selected by the designer and $\theta=(\theta_{1},\dots,\theta_{K})\in\mathbf{R}^{K}$ is a parameter vector. The parameters are decided by solving a finite-dimensional analogue of Problem (OC-$Q$), with $Q$ replaced by $Q_{\theta}$:

[TABLE]

Convex constraints on $\theta$ can also be added, e.g., to cultivate a particular structure in the control policy. Given $Q_{\theta}$, the unique corresponding causal output feedback policy can be constructed from the recursion in Theorem 2.

Because $Q_{\theta}$ is affine in $\theta$ and convexity is preserved under composition with an affine mapping, the $Q$-design problem (OC-$Q_{\theta}$) inherits the convexity of Problem (OC-$Q$) for convex systems. It follows that $Q$-design can be applied to convex systems, even nonlinear ones, using finite-dimensional convex optimization. Risk measures in Problem (OC-$Q_{\theta}$) can be approximated if necessary, as discussed in §V-B.

In principle, $Q$-design can solve the optimal control problem (OC-$Q$) to any degree of accuracy for a sufficiently large basis. In practice, $Q$-design has two disadvantages. First, its performance depends intimately on the choice of basis policies. In [42], Skaf and Boyd explore the natural choice of affine policies in the context of linear systems. While affine $Q$-design performs well in a number of problems, the performance is typically not as good as open-loop MPC. To our knowledge, the general problem of finding good nonlinear basis policies has not yet been solved. A second disadvantage of $Q$-design is that, as discussed in §IV-A, it can be applied only to purifiable systems. If the system is not purifiable, then an implementable output feedback policy may not be recoverable from the $Q$-designed policy.

V-A3 Closed-loop model predictive control

The final approximate solution method we discuss, closed-loop MPC, can be viewed as receding horizon $Q$-design. As in open-loop MPC, in closed-loop MPC we solve an optimal control subproblem at each time step over a truncated, receding horizon. The key difference is that the closed-loop MPC subproblems include a recourse model, i.e., a model of the controller’s response to future information as it becomes available. More precisely, the open-loop MPC subproblems are truncated versions of the open-loop optimal control problem (OLOC), while the closed-loop MPC subproblems are truncated versions of the $Q$-design problem (OC-$Q_{\theta}$). Closed-loop MPC with affine recourse models is developed for linear systems in [42, 47, 50, 51, 56, 57].

Closed-loop MPC addresses the two disadvantages of $Q$-design. Like open-loop MPC, closed-loop MPC can be applied to systems that are not purifiable. To see this, we note that the first step of the recursion in Theorem 2,

[TABLE]

can always be implemented, even if later steps cannot. Closed-loop MPC also tends to be less sensitive than $Q$-design to the choice of basis policies; in closed-loop MPC the policy designed at each time step is merely a model of future recourse, while in $Q$-design the policy is the actual source of the implemented control inputs.

Closed-loop MPC can outperform open-loop MPC due to the inclusion of a recourse model. [42] Its design process is similarly straightforward; the only additional step is choosing basis policies. Like open-loop MPC, however, closed-loop MPC involves computationally intensive, online optimization. A closed-form expression for the closed-loop MPC policy is generally not available.

V-B Risk measure approximation

The control methods discussed in §V-A all require solving finite-dimensional convex optimization subproblems under uncertainty. These subproblems can be solved exactly in a few special cases. In general, however, we must resort to approximate solution methods. [3, 6, 10]

We now describe one approximate solution method based on sampling. This method can accommodate each of the risk measures discussed in §II-B. We begin by obtaining samples $\delta^{(1)},\dots,\delta^{(N)}$ from the distribution of the exogenous input trajectory $\delta$, or in the robust setting, from the corresponding uncertainty set. Samples could be obtained, e.g., from historical data or a pseudorandom number generator. Each risk measure $\mathcal{R}_{j}$ is then replaced by an approximation $\hat{\mathcal{R}}_{j}^{N}$ based on the samples $\delta^{(1)},\dots,\delta^{(N)}$. This results in a convex optimization problem (with random input data) that can be solved using off-the-shelf software. Depending on the underlying risk measures, this sample-based approximate solution scheme is known as scenario optimization [77, 78, 79, 80, 81, 82] or sample-average approximation [6, 86, 87, 88].

Table II shows sample-based approximations for the five risk measures in Table I. The worst-case value and value-at-risk (which generate robust and chance constraints, respectively) can be approximated by maxima over the full sample. Expectation-type risk measures, including conditional value-at-risk, can be approximated using sample averages. Both of these approximations preserve the convexity of the underlying cost or constraint function. Theoretical bounds on violation probabilities for robust and chance constraints are available. [77, 78, 79, 80, 81, 82] Some convergence results for sample-average approximation can be found in [6, 86, 87, 88]. Variance reduction methods and hypothesis tests of solution quality can also be applied. [6]

To illustrate this method, we consider the open-loop optimal control problem (OLOC). The sample-based approximation to this problem is to

[TABLE]

The $Q$-design problem (OC-$Q_{\theta}$) can be approximated similarly. The decision variables in these problems ($\tilde{u}$ and $\theta$, respectively) can be regularized to avoid overfitting the training data $\delta^{(1)},\dots,\delta^{(N)}$. The computational complexity of the sampled problem generally scales linearly with $N$, so relatively large sample sizes can often be used. Different sample sizes $N_{0},\dots,N_{J}$ can be used to approximate the different risk measures $\mathcal{R}_{0},\dots,\mathcal{R}_{J}$.

VI Numerical example

In this section, we demonstrate the new capabilities developed in this paper through a numerical example. The example includes nonlinear dynamics, non-additive uncertainty and the ${\rm CVaR}_{\beta}$ risk measure.

We consider a single-input, single-output system with dynamics

[TABLE]

We assume the controller has perfect state information ($y_{t}=x_{t}$), so that we can directly compare control methods in isolation from the state estimation problem. The unforced dynamics have a stable equilibrium at the origin and an unstable equilibrium at unity. Our primary control objective is to maintain the state within the basin of attraction of the stable equilibrium, i.e., to satisfy the constraint $c_{1}(x)\leq 0$, where

[TABLE]

We accept that this constraint may occasionally be violated due to the uncertain initial state and disturbances, but tolerate violations only if they are both infrequent and small. For this reason, we measure the risk of constraint violations by the conditional value-at-risk ($\mathcal{R}_{1}={\rm CVaR}_{\beta}$) at confidence level $\beta=0.95$. This risk measure addresses both the probability and (conditionally) expected magnitude of constraint violations. As discussed in §II-B, ${\rm CVaR}_{\beta}c_{1}(x)$ can be constrained by minimizing a scalar $\alpha$ subject to

[TABLE]

We would also like to minimize the total cost of control effort,

[TABLE]

where $p(\delta_{1}),\dots,p(\delta_{T})\geq 0$ are uncertain prices. We choose the cost risk measure $\mathcal{R}_{0}=\mathop{{\bf E{}}}$.

The dynamics (7), while nonlinear, are nondecreasing in $x_{t}$ and convex in $(x_{t},u_{t})$ for all $\delta_{t+1}$. (We assume $w(\cdot)\leq 20$, so the coefficient multiplying $(x_{t})_{+}^{2}$ is nonnegative.) The cost function $c_{0}$ is convex in $u$ for all $\delta$, and the constraint function $c_{1}$ is convex and nondecreasing in each of $x_{1},\dots,x_{T}$. By Theorem 1, therefore, the system is convex. It is also purifiable, since the controller has perfect state information and the dynamics are invertible in the disturbance $w(\delta_{t+1})$. The purified output at time $t$ is $\xi_{t}(\delta_{0:t})=w(\delta_{t})$, with purifier

[TABLE]

The inverse purifier $q_{t}$ can be straightforwardly constructed from the system dynamics. Theorem 2 therefore applies, establishing a bijection between disturbance feedback policies and state feedback policies for this system. This gives our (infinite-dimensional, convex) control design problem:

[TABLE]

We compare five suboptimal controllers for this system. The first is the optimal affine disturbance feedback policy, computed via the $Q$-design procedure. The second is a nonlinear $Q$-designed policy with piecewise quadratic basis policies tuned through trial and error. The third is the certainty-equivalent variant of open-loop MPC discussed in §V-A1; subproblems are solved under a single prediction of the exogenous input trajectory (in this case, its conditional expectation). The fourth is open-loop MPC with the same risk measures as Problem (8). The fifth is closed-loop MPC with the same risk measures and an affine disturbance feedback recourse model. We refer to the fourth and fifth controllers as open-loop and closed-loop scenario MPC, respectively, as is common in the literature. [51, 89]

Implementing each of these controllers involves numerically solving subproblems generated by Problem (8). As discussed in §V-B, we replace the expected values in these subproblems by sample averages over a training set of 1,000 sample exogenous input trajectories. We use a validation set of another 1,000 samples to tune parameters such as the basis policies in nonlinear $Q$-design and the MPC prediction horizon. We then compare the policies’ performance in a test set of another 2,000 samples.

In simulations, the initial state has a half-normal distribution on $[1,\infty)$ with mean two. The disturbances $w(\delta_{1}),\dots,w(\delta_{T})$ are identically Beta$(2,4)$ distributed, shifted and scaled to have mean zero and support $[-1,2]$. Each price $p(\delta_{t})$ of control effort is exponentially distributed with mean $2+\cos(2\pi t/T)$; prices are lowest (on average) in the middle of the control horizon. The prices, disturbances and initial state are mutually independent.

Optimization is done in MATLAB using the Gurobi solver and the CVX modeling toolbox [90], which makes specifying our problems very easy. For example, the following code computes the optimal open-loop control input trajectory over $N$ sample exogenous input trajectories stored in x0 ($1\times N$), w ($T\times N$) and p ($T\times N$):

variables a u(T,1) expression x(T,N) x = stateTrajectories(x0,u,w,f,T); minimize( a + sum(c0(u,p))/N ) subject to sum(pos(c1(x) - a))/N <= -a*(1 - b)

We wrote the vectorized function stateTrajectories to play the role of the input-state mapping $\phi$, recursively building the state trajectory in each of the $N$ Monte Carlo simulations from the dynamics function f. It typically speeds up CVX modeling by an order of magnitude compared to a loop over Monte Carlo runs. This function, along with all other code used in this paper, is online at [91].

Figure 2 shows the sample-average cost and constraint conditional value-at-risk of each controller in the test set of exogenous input trajectories. Table III shows the frequency with which each controller satisfies the constraint $x_{t}\leq 1$ in the test set. The five controllers strike different trade-offs between cost and robust constraint satisfaction. Certainty-equivalent MPC is aggressive, achieving low cost but poor robustness. The optimal affine controller satisfies constraints more robustly at the cost of increased control effort. The $Q$-designed nonlinear controller performs significantly better than the optimal affine controller, achieving similar cost and robustness to both of the scenario MPC variants. Including an affine recourse model in scenario MPC reduces cost slightly over open-loop scenario MPC, but also increases the risk of constraint violation. In this example, the robustness improvement of scenario MPC over certainty-equivalent MPC comes mainly from the use of samples, rather than the recourse model.

This numerical example demonstrates the key message of this paper: Suboptimal control methods that perform well for linear convex systems can be applied directly to nonlinear convex systems using the same software, and can perform similarly well. In this example, the $Q$-designed and predictive controllers show similar performance trends to the trends demonstrated for linear systems in [41, 42] and elsewhere. In particular, (a) scenario MPC is more robust than certainty-equivalent MPC, while achieving lower cost than affine controllers, and (b) nonlinear $Q$-designed controllers with good basis policies can be competitive with scenario MPC.

VII Conclusion

In this paper, we explored the class of nonlinear systems for which optimal control design under uncertainty can be cast as a convex optimization problem. We adopted a flexible approach to the risks associated with constraint violations and excessive costs, accommodating both the robust and stochastic views. We showed that open-loop optimal control is convex for nonlinear systems with convex or concave dynamics, provided the dynamics, cost and constraint functions satisfy certain monotonicity properties. We then showed that under the same conditions, closed-loop control design can be reformulated as a convex program if, in addition, the measured outputs can be reversibly ‘purified’ of the influence of the past control inputs.

The practical value of these results is the guarantee that, for a class of nonlinear systems, the subproblems solved in various suboptimal control methods are convex. This removes concerns about solvers getting stuck in local minima, and enables the use of convenient modeling software and reliable, efficient solvers. We illustrated this numerically for two methods, the $Q$-design procedure and model predictive control.

There are a number of opportunities to extend this work. First, it would be interesting to survey applications in which nonlinear convex systems arise. We are aware of several examples, including cancer and HIV treatment scheduling [75], voltage control in electricity distribution networks [75], freeway traffic congestion management [76] and energy storage control [84, 85], but this list is likely incomplete.

Second, the requirement of purifiability appears to significantly restrict the uncertainty structures and nonlinearities admissible for $Q$-design. Exploring purifiability further could enable convex control design for a richer class of nonlinear systems. We note, however, that other methods, notably including model predictive control, do not require purifiability.

Third, the class of nonlinear systems for which optimal control is convex could be further explored. We characterized only a subset of this class of systems; our method relied on the application of simple composition rules. While this method is constructive and compatible with convenient modeling software such as CVX [90] and YALMIP [92], it is not exhaustive.

Appendix A Proof of Theorem 1

Our task is to show that the function

[TABLE]

is convex for all $\delta$. Here we have expressed the state trajectory as

[TABLE]

We will define the mappings $C$, $v$ and $\eta$ shortly.

We recall that convexity is preserved under composition with an affine mapping and under the composition of a convex nondecreasing function with a convex function [83], and that $c_{j}$ is nondecreasing in each element of $x^{\rm cvx}$ by assumption. It therefore suffices to show that each row of $\eta(\cdot,\delta)$ is convex for all $\delta$.

To do this, we need the recursive definitions of $C$, $v$ and $\eta$. They are initialized by

[TABLE]

For $t=2,\dots,T$,

[TABLE]

The proof now proceeds by induction. At $t=1$, $\eta_{1}$ is convex in $u_{0}$ by the convexity of $h_{1}$. For the inductive step, we suppose that each row of $\eta_{t-1}$ is convex in $u_{0:t-2}$. By assumption, each row of $h_{t}$ is nondecreasing in $x^{\rm cvx}_{t-1}$ and convex. Therefore, $\eta_{t}$ involves compositions of convex functions with the affine mapping $u_{0:t-2}\mapsto C_{t-1}(\delta_{t-1})u_{0:t-2}+v_{t-1}(\delta_{0:t-1})$ and of convex nondecreasing functions with convex functions, namely the rows of $u_{0:t-2}\mapsto\eta_{t-1}(u_{0:t-2},\delta_{0:t-1})$. Applying our two composition rules, we see that each row of $\eta_{t}$ is convex. This concludes the inductive step and the proof.

Appendix B Proof of Theorem 2

Our task is to show that purifiability implies a bijection exists between causal $\pi$ in $y$ and causal $Q$ in some purified output that depends on $\delta$ alone. To do this, we suppose the system is purifiable. We build a causal $\pi$ in $y$ from a given causal $Q$ in the purified output

[TABLE]

We recall that $e_{0}=y_{0}$, always. The construction of $\pi$ from $Q$ is recursive:

[TABLE]

Supposing this holds for $t=0,\dots,\tau-1$, at $t=\tau$ we have

[TABLE]

By induction, therefore, the recursion (9) holds for all $t=0,\dots,T-1$. Given a causal $\pi$ in $y$, a causal $Q$ in $e$ can be constructed similarly from the recursion

[TABLE]

This establishes the bijection and concludes the proof.

Acknowledgments

The authors thank Anders Rantzer, Andy Ruina and Eilyan Bitar for their helpful comments, and gratefully acknowledge support from the National Science Foundation under grant 1711546.