Model predictive control with stage cost shaping inspired by reinforcement learning

TL;DR

This paper investigates a model predictive control method that incorporates reinforcement learning principles to shape the stage cost, aiming to analyze its suboptimality and compare its performance to the infinite-horizon optimal control.

Contribution

It introduces a novel stage cost shaping approach inspired by reinforcement learning within the MPC framework and provides a suboptimality analysis relating it to the infinite-horizon value function.

Findings

Suboptimality bounds derived for the proposed MPC variant.

Case studies demonstrate effectiveness across various initial conditions.

Stage cost shaping improves control performance in tested systems.

Abstract

This work presents a suboptimality study of a particular model predictive control with a stage cost shaping based on the ideas of reinforcement learning. The focus of the suboptimality study is to derive quantities relating the infinite-horizon cost function under the said variant of model predictive control to the respective infinite-horizon value function. The basis control scheme involves usual stabilizing constraints comprising of a terminal set and a terminal cost in the form of a local Lyapunov function. The stage cost is adapted using the principles of Q-learning, a particular approach to reinforcement learning. The work is concluded by case studies with two systems for wide ranges of initial conditions.

Tables

Table 1. TABLE I: Procedure of the online scheme.

Algorithm:

1) Initialize: Choose $F$, $\mu$, ${𝕏}_f$ and $c_0\left(i\right)\in {ℝ}_{>0}$, $i=0,\dots ,N-1$ Set $k=0$. Then, 2) Obtain state $x_k$. 3) Solve (8) using $c_k\left(i\right)$, $i=0,\dots ,N-1$, to obtain ${\left\{u_k^{\ast }\left(i\right)\right\}}_{i=0}^{N-1}$ and ${\left\{x_k^{\ast }\left(i\right)\right\}}_{i=0}^N$ 4) Solve (13) under the constraint set ${𝒲}_k$ to obtain $c_{k+1}\left(i\right)\in {𝒲}_k$, $i=0,\dots ,N-1$ 5) Apply the first element of the sequence $u_k^{\ast }\left(0\right)$ to (1) 6) $k\mapsto k+1$, go to 2)

Equations

$$x_{k+1}=f(x_k,u_k),k=0,1,\dots ,x_0\in \mathbb{X},\text{vspace}\ast -.5em$$

$$\begin{array}{rl}\forall x_0\in \mathbb{X}:V^{\infty }(x_0):=\underset{\{u_k{\}}_{k=0}^{\infty }}{\min }& \sum _{k=0}^{\infty }l(x_k,u_k),\text{s.t.}\text{eqref}eq:sys\end{array}$$

$${\tilde{V}}_{\text{MPC}}^{\infty }(x_k)$$

$$\underset{\{u_k(i){\}}_{i=0}^{N-1}}{\min }$$

$$x_k(i+1)=f(x_k(i),u_k(i)),x_k(0)=x_k$$

$$x_k(i)\in \mathbb{X},u_k(i)\in \mathbb{U}$$

$$x_k(N)\in {\mathbb{X}}_f.$$

$$\forall (x,u)\in \mathbb{X}\times \mathbb{U}:\mathcal{Q}(x,u)=l(x,u)+\underset{a\in \mathbb{U}}{\min }\mathcal{Q}(f(x,u),a),$$

$$\forall x_k\in {\mathbb{X}}_f:{\Delta }_{{\mu }_F(x_k)}F(x_k)\le -{\alpha }_F(x_k).$$

$$\forall x_k\in {\mathbb{X}}_f:{\Delta }_{{\mu }_F(x_k)}F(x_k)=-{\bar{\alpha }}_{k}l(x_k,{\mu }_F(x_k)),$$

$$\forall k\in {\mathbb{N}}_0:{\Delta }_{\mu ({\tilde{x}}_k)}W({\tilde{x}}_k)\le -c_{k}l({\tilde{x}}_k,\mu ({\tilde{x}}_k)),$$

$$\sum _{k=0}^{\infty }l({\tilde{x}}_k,\mu ({\tilde{x}}_k))\le V^{\infty }(x_0)+{\delta }_0-\sum _{k=0}^{\infty }(\Delta l{)}_k,$$

$$\sum _{k=0}^{\infty }(\Delta l{)}_k\le {\delta }_0.$$

$$W({\tilde{x}}_{k+M+1})-W({\tilde{x}}_k)\le$$

$$=$$

$$\sum _{i=0}^{\infty }l({\tilde{x}}_i,\mu ({\tilde{x}}_i))$$

$$\iff \underset{<\infty }{\underbrace{\sum _{i=0}^{\infty }l({\tilde{x}}_i,\mu ({\tilde{x}}_i))}}$$

$$0\le \sum _{i=0}^{\infty }l({\tilde{x}}_i,\mu ({\tilde{x}}_i))-V^{\infty }(x_0)$$

$$\Rightarrow \sum _{i=0}^{\infty }(\Delta l{)}_i$$

$${\tilde{V}}^{\infty }(x_k^{\ast },k)$$

$$\underset{\{u_k(i){\}}_{i=0}^{N-1}}{\min }$$

$$x_k(i+1)=f(x_k(i),u_k(i)),x_k(0)=x_k$$

$$x_k(i)\in \mathbb{X},u_k(i)\in \mathbb{U}$$

$$x_k(N)\in {\mathbb{X}}_f\subset \mathbb{X}.$$

$$\forall k\in {\mathbb{N}}_0:{\Delta }_{u_k^{\ast }(0)}{\tilde{V}}^{\infty }(x_k^{\ast },k)\le -c_k(0)l(x_k^{\ast },u_k^{\ast }(0)),$$

$$\begin{array}{rl}& {\mathcal{W}}_{k-1}:=\{C(i)\in {\mathbb{R}}_{>0},i=0,\dots ,N-1:\\ & \sum _{i=1}^{N-1}\left(C(i-1)-c_{k-1}(i)\right)l(x_{k-1}^{\ast }(i),u_{k-1}^{\ast }(i))\\ & +F(f(x_{k-1}^{\ast }(N),{\mu }_F(x_{k-1}^{\ast }(N))))-F(x_{k-1}^{\ast }(N))\\ & +C(N-1)l(x_{k-1}^{\ast }(N),{\mu }_F(x_{k-1}^{\ast }(N)))\le 0\},\end{array}$$

$$\sum _{i=1}^{N-1}{c}_{k+1}(i-1)l(x_k^{\ast }(i),u_k^{\ast }(i))$$

$$+c_{k+1}(N-1)l(x_k^{\ast }(N),{\mu }_F(x_k^{\ast }(N)))$$

$$+F(f(x_k^{\ast }(N),{\mu }_F(x_k^{\ast }(N))))$$

$$\le$$

$$=$$

$${\tilde{V}}^{\infty }(x_{k+1}^{\ast },k+1)$$

$$\le$$

$$\forall x\in \mathbb{X}:V^{\infty }(x)=\underset{u\in \mathbb{U}}{\min }\mathcal{Q}(x,u),$$

$$\displaystyle\begin{split}w_{j}=&\operatorname*{arg\,min}_{C}\;\big{(}l(x,u)-Cl(x,u)\\[-5.0pt] &\hskip 73.97733pt+\min_{a\in\mathbb{U}}\;w_{j-1}l(f(x,u),a)\big{)}^{2},\end{split}$$

$$\displaystyle\begin{split}c_{k+1}(i)=&\operatorname*{arg\,inf}_{C\in\mathbb{R}_{>0}}\;\big{(}l(x_{k}^{\ast}(i),u_{k}^{\ast}(i))-Cl(x_{k}^{\ast}(i),u_{k}^{\ast}(i))\\[-5.0pt] &\hskip 51.21504pt+c_{k}(i)\min_{a\in\mathbb{U}}\;l(f(x_{k}^{\ast}(i),u_{k}^{\ast}(i)),a)\big{)}^{2},\end{split}$$

$$C^{+}=\underset{[C_0,\dots ,C_{N-1}{]}^{\top }\in {\mathbb{R}}_{>0}^N}{\mathrm{arginf}}\sum _{i=0}^{N-1}({\beta }_i-C_{i}l(x_k^{\ast }(i),u_k^{\ast }(i)){)}^2$$

$$c_{k+1}(i-1)=c_k(i),i=1,\dots ,N-1.$$

Full text

I Introduction

Briefly, MPC converts an infinite-horizon potential control problem to tractable, finite-horizon, online optimizations. Since several control setups are available for the corresponding finite-horizon optimization, different performances with respect to the original infinite-horizon problem can be expected. In particular, unconstrained MPC in the sense of minimizing solely a truncation of the infinite-horizon cost each time step without any stabilizing so-called terminal constraints was studied rigorously [9, 5, 16]. Though the question of suboptimality under stabilizing terminal constraint was addressed in various setups [21, 6], they require preliminary assumptions on the LF and/or the terminal set, of which some are reviewed and relaxed herein.

Problem formulation. Consider in the following the discrete-time system

[TABLE]

with state $x_{k}\in\mathbb{X}\subset\mathbb{R}^{n_{x}}$ and control $u_{k}\in\mathbb{U}\subset\mathbb{R}^{n_{u}}$. The map $f:\mathbb{X}\times\mathbb{U}\rightarrow\mathbb{X}$ is continuous in both arguments and $(\bar{x},\bar{u})=(0,0)$ is the unique equilibrium point , i. e., $f(x,u)=x$ if and only if $x=\bar{x}$ and $u=\bar{u}$.

In this work, deterministic and fully available dynamics are considered, whereas there exist approaches of adapting the model , e. g., by neural networks [13].

With (1), the infinite-horizon optimal control problem (IHOCP) amounts to solving

[TABLE]

where the stage cost (or state-input penalty) has the form $l:\mathbb{X}\times\mathbb{U}\rightarrow\mathbb{R}_{\geq 0}$, with $l(\bar{x},\bar{u})=0$, $l(x,u)>0$ for $(x,u)\neq(\bar{x},\bar{u})$ and $l(x,\bar{u})\leq l(x,u)$ for any $x$ and $u$. Denote $\mathbb{U}_{\text{ad}}$ the admissible set, containing control sequences $\{u_{k}\}_{k=0}^{\infty}$, with $u_{k}\in\mathbb{U}$ and $u_{k}\rightarrow\bar{u}$ for $k\rightarrow\infty$, for which $f(x_{k},u_{k})\in\mathbb{X}$, $f(x_{k},u_{k})\rightarrow\bar{x}$ for $k\rightarrow\infty$ and $\sum_{k=0}^{\infty}l(x_{k},u_{k})<\infty$. It is assumed that $\mathbb{U}_{\text{ad}}\neq\varnothing$. The minimizer $\{u_{k}^{\infty}\}_{k=0}^{\infty}\in\mathbb{U}_{\text{ad}}$ associated with (2) may be given in a closed form as a feedback , i. e., there exists a function $\mu^{\infty}:\mathbb{X}\rightarrow\mathbb{U}$ such that $u_{k}^{\infty}=\mu^{\infty}(x_{k})$.

Let $F:\mathbb{X}_{f}\rightarrow\mathbb{U}$ and $\mathbb{X}_{f}\subset\mathbb{X}$ containing $\{\bar{x}\}$ in the interior. A nonlinear MPC (NMPC) optimization problem for time $k$, with terminal cost and terminal set constraint, may be given by

[TABLE]

Constraints in state and input in (3c) are commonly included whereas (3d) is the so-called stabilizing constraint. Certain assumptions on the terminal cost $F$ – namely, $F$ being a local LF with controller $\mu_{F}$ such that $x\in\mathbb{X}_{f}\Rightarrow f(x,\mu_{F}(x))\in\mathbb{X}_{f}$ and $\Delta_{\mu_{F}(x)}F(x)$ having a specific decay rate – allows relating $\tilde{V}_{\text{MPC}}^{\infty}(x_{k})$ to the IHOCP.

The main drawback of this approach is that the state sequence is forced to converge to $\mathbb{X}_{f}$ after $N$ steps, which may render the minimizer of (3) possibly suboptimal with respect to the first $N$ elements of the infinite sum in (2).

Q-learning-inspired stage cost shaping. The aim of this work is to investigate a particular adaptation scheme of the stage cost in the MPC scheme (3) in the form $l\mapsto c\cdot l$. This specific approach is motivated by the fact that online adaptation or learning, in general, may use only limited or single (the current) state-input values $(x_{k},u_{k})$. Although “buffers”, as it was suggested in [12], can be utilized, the presented adaptation scheme uses solely the current data. Other methods of incorporating learning into MPC include data-based constraint design for robust MPC [23] or terminal cost adaptation in MPC through means of iterative learning control [17]. The adaptation scheme is based on the principles of Q-learning as follows. Interpreting $c\cdot l$ as an approximation to the Q-function

[TABLE]

the coefficients $c$ should be updated so as to minimize the temporal difference [20]. More details on this approach are given in the subsequent sections. The aim of the work is to derive suboptimality bounds of the respective MPC scheme under such an adaptation of the stage cost. Particular realizations of the coefficient update rule are discussed in Section II-C. The case studies of Section IV present comparison of the baseline MPC with the suggested stage-cost-shaping MPC for wide ranges of initial conditions.

Notation. The natural numbers including zero are denoted by $\mathbb{N}_{0}:=\mathbb{N}\cup\{0\}$. For any scalar functions $W_{1}(x_{k},k):\mathbb{R}^{n_{x}}\times\mathbb{N}_{0}\rightarrow\mathbb{R}_{\geq 0}$ and $W_{2}(x_{k}):\mathbb{R}^{n_{x}}\rightarrow\mathbb{R}_{\geq 0}$, denote $\Delta_{u}W_{1}(x_{k},k):=W_{1}(f(x_{k},u),k+1)-W_{1}(x_{k},k)$ and $\Delta_{u}W_{2}(x_{k}):=W_{2}(f(x_{k},u))-W_{2}(x_{k})$ the difference of subsequent function values along the state recursion (1) under some $u\in\mathbb{R}^{n_{u}}$.

II Suboptimality Description and Algorithm

In the following, some key assumptions on the function $F$ are introduced, one of which can be found frequently in MPC literature [4, 8]. Furthermore, optimality and stability related statements are presented, summarizing the objective of this work. Specifically, a suboptimality comparison is suggested for the case of the predictive scheme under parametrized decay of a local LF (3).

II-A Central Assumptions and Suboptimality Estimate

Essentially, a key ingredient to ensure stability is the existence of a local controller inside the set, which the terminal predicted state is sought to reach. This means:

Assumption 1

There exists a local controller $\mu_{F}:\mathbb{X}_{f}\rightarrow\mathbb{U}$ and a local LF $F:\mathbb{X}_{f}\rightarrow\mathbb{R}_{\geq 0}$ such that

[TABLE]

It is not until later in the analysis that the function $F(x)$ is required to have a decay of magnitude of the stage cost [2, 7, 8]:

Assumption 2

There exists a local controller $\mu_{F}:\mathbb{X}_{f}\rightarrow\mathbb{U}$ and a local LF $F:\mathbb{X}_{f}\rightarrow\mathbb{R}_{\geq 0}$ such that

[TABLE]

where $1\leq\bar{\alpha}_{k}<\infty$, for all $k\in\mathbb{N}_{0}$.

The function $F$ in Assumption 2 can be regarded as the infinite tail inside $\mathbb{X}_{f}$ under the stage cost $l$, obtained , e. g., by local LQ analysis [2].

Proposition 1

Let $W:\mathbb{X}\rightarrow\mathbb{R}_{\geq 0}$ be a LF on $\mathbb{X}$ , i. e., $W(\bar{x})=0$ and positive for any $x\in\mathbb{X}-\{\bar{x}\}$, and $\mu:\mathbb{X}\rightarrow\mathbb{U}$ the control policy, with $\mu(\bar{x})=\bar{u}$ and $\{\mu(\tilde{x}_{k})\}_{k=0}^{\infty}\in\mathbb{U}_{\text{ad}}$, where $\tilde{x}_{k+1}=f(\tilde{x}_{k},\mu(\tilde{x}_{k}))$ under some $\tilde{x}_{0}\in\mathbb{X}$. Furthermore, let $W$ and $\mu$ be such that

[TABLE]

for some sequence $\{c_{k}\}_{k=0}^{\infty}$, with $c_{k}\in(0,\bar{c})$, $0<\bar{c}<\infty$. Denote $\delta_{0}=W(x_{0})-V^{\infty}(x_{0})$ for any $x_{0}\in\mathbb{X}$ and $(\Delta l)_{k}:=c_{k}l(\tilde{x}_{k},\mu(\tilde{x}_{k}))-l(\tilde{x}_{k},\mu(\tilde{x}_{k}))$. It holds that

[TABLE]

where furthermore

[TABLE]

Proof:

By (5), for any $k,M\in\mathbb{N}_{0}$, $M\geq k$,

[TABLE]

With $\tilde{x}_{k+M+1}\rightarrow\bar{x}$ and thus $W(\tilde{x}_{k+M+1})\rightarrow 0$ for $M\rightarrow\infty$, and $k=0$,

[TABLE]

using $\delta_{0}$ as defined. By optimality, $V^{\infty}(x_{0})$ is smaller or equal the value on the left-hand side in the above inequality. Therefore, the sum over $(\Delta l)_{i}$ is bounded by $\delta_{0}$ as

[TABLE]

Since $\tilde{x}_{i}\rightarrow\bar{x}$, $l(\tilde{x}_{i},\mu(\tilde{x}_{i}))\rightarrow 0$ for $i\rightarrow\infty$, and thus $(\Delta l)_{i}\rightarrow 0$. With the boundedness of $c_{i}\leq\bar{c}$, the limit of the sequence of partial sums of the left-hand side in the above inequality is a real number and the infinite series is convergent and finite. ∎

If $W(x)\equiv V^{\infty}(x)$, (5) changes to an equation with $c_{k}=1$, for all $k\in\mathbb{N}_{0}$, and thus $(\Delta l)_{k}=\delta_{0}=0$. Assuming that (in general) $\delta_{0}\geq 0$, (7) motivates to search for such functions that yield (5) with $c_{k}\geq 1$ values. Roughly speaking, the above states that the initial discrepancy $\delta_{0}>0$ may be partially compensated for by increasing the decay.

II-B Setup of the Receding Horizon Scheme

For a given sequence $c_{k}(i)\in\mathbb{R}_{>0}$, $i=0,\dotsc,N-1$, define the following value function at time step $k$ for a state $x_{k}^{\ast}\in\mathbb{X}$

[TABLE]

The state sequence associated with the minimizing control sequence $\{u_{k}^{\ast}(0),\dotsc,u_{k}^{\ast}(N-1)\}$ to (8) is denoted by $\{x_{k}^{\ast}(1),\dotsc,x_{k}^{\ast}(N)\}$, whereas $x_{k+1}^{\ast}=x_{k+1}=f(x_{k},u_{k}^{\ast}(0))$ is the state along (1) under the first element of the control sequence, with $x_{k}^{\ast}=x_{k}=x_{k}^{\ast}(0)$. It is assumed in the following that such a minimizing sequence to (8) exists at time $k=0$. Due to the time-invariance of the dynamics (1) as well as the constraints (8c) and (8d), the control action is recursively feasible for subsequent times [14].

It can be shown that a decay of the form

[TABLE]

can be obtained through one inequality constraint over all coefficients in (8). That is, the coefficients must lie in the set $\mathcal{W}_{k-1}\ni c_{k}(i)$, $i=0,\dotsc,N-1$, defined by

[TABLE]

which contains the coefficients, controls and states to (8) of preceding time step $k-1$ (associated with $\tilde{V}^{\infty}(x_{k-1}^{\ast},k-1)$, and thus denoted with the suffix $k-1$).

For $k=0$, certain initial coefficients $c_{0}(i)\in\mathbb{R}_{>0}$ are used in $\tilde{V}^{\infty}(x_{0},0)$, which may be determined by some offline optimization or chosen “suitably” (see Section III-B for a discussion).

Proposition 2

Let Assumption 1 hold. Let $c_{0}(i)\in\mathbb{R}_{>0}$, $i=0,\dotsc,N-1$. If for all $k\in\mathbb{N}$ and all $i\in\{0,\dotsc,N-1\}$, $c_{k}(i)\in\mathcal{W}_{k-1}$, then (9) holds for all $k\in\mathbb{N}_{0}$.

Proof:

Consider the inequality (10) in $\mathcal{W}_{k-1}$ at time $k-1\mapsto k$. Adding $\sum_{i=1}^{N-1}c_{k}(i)l(x_{k}^{\ast}(i),u_{k}^{\ast}(i))$ and $F(x_{k}^{\ast}(N))$ to both sides reads

[TABLE]

Define the feasible sequences $\{\tilde{x}_{k+1}(0),\dotsc,\tilde{x}_{k+1}(N)\}:=\{\{x_{k}^{\ast}(1),\dotsc,x_{k}^{\ast}(N)\},f(x_{k}^{\ast}{N},\mu_{F}(x_{k}^{\ast}(N)))\}$ and $\{\tilde{u}_{k+1}(0),\dotsc,\tilde{u}_{k+1}(N-1)\}:=\{u_{k}^{\ast}(1),\dotsc,u_{k}^{\ast}(N-1),\mu_{F}(x_{k}^{\ast}(N))\}$. Substituting this into the left-hand side of the inequality gives

[TABLE]

by optimality of $\tilde{V}^{\infty}(x_{k+1}^{\ast},k+1)$. ∎

Remark 1

By the intermediate calculation of coefficients satisfying (10), at each time $k$ after solving (8), stability can be enforced using a local LF with any decay $\alpha_{F}$. That is, no stage cost related minimum decay rate is required inside the terminal set $\mathbb{X}_{f}$. This reduces Assumption 2 to the more general Assumption 1.

Recall from , e. g., [10], that for a time-variant LF function $W(x,k)$ with controller $\mu(x)$ there exist $\alpha_{1},\alpha_{2}\in\mathcal{K}_{\infty}$ and $\alpha_{3}\in\mathcal{K}$ such that for all $x\in\mathbb{X},\,\,k\in\mathbb{N}_{0}$, $\alpha_{1}(\|x\|)\leq W(x,k)\leq\alpha_{2}(\|x\|)$ and $\Delta_{\mu(x)}W(x,k)\leq-\alpha_{3}(\|x\|)$. Thus, for stability, the value function in (8) must be bounded, which is equivalent to bounded coefficients for non-zero and bounded stage costs.

II-C Coefficient Update

In order to obtain somewhat meaningful coefficients $c_{k}(i)$ and respective update laws, aspects of Q-learning [22], and more specifically the Q-function, can be transferred into the above context. That is, for updating the coefficients, considering algorithms from Q-learning may be beneficial since an adaptation to the optimal decay rate is sought that minimizes the difference between the terms in (7). The Q-function $\mathcal{Q}:\mathbb{X}\times\mathbb{U}\rightarrow\mathbb{R}_{\geq 0}$, formally defined as in (4), is commonly replaced by some parametric architecture $Q(x,u):=\langle w^{\ast},\varphi(x,u)\rangle$, where $w^{\ast}\in\mathbb{R}^{p}$ are the parameters to a regressor $\varphi:\mathbb{X}\times\mathbb{U}\rightarrow\mathbb{R}^{p}$, comprising some basis functions. Since [11]

[TABLE]

the function $Q$ seeks to approximate the infinite-horizon optimal value.

In order to transfer this approach to the (online) predictive scheme, the stage cost may be used as regressor. The underlying hypothesis in choosing particularly $\varphi(x,u)\equiv l(x,u)$ , i. e., $Q(x,u)\equiv w^{\ast}\,l(x,u)$, is that the optimal value function $V^{\infty}$ may be expressed as a certain magnitude, or factor, of the stage cost $l$.

Temporal difference update. For computation of $w^{\ast}$, certain offline iterative routines are available [1], approximating $V^{\infty}$ a priori over samples in the state-input space. The temporal difference method [20] is one particular approach that can be used to compute $w^{\ast}$ by iterating through

[TABLE]

$j=0,1,\dotsc$, until $\|w_{j}-w_{j-1}\|\leq\varepsilon_{w}$ for all $x,u$ (samples) and some $\varepsilon_{w}>0$. Then, take $w^{\ast}:=w_{j}$.

Remark 2

The above update rule refers to an off-policy learning as “hypothetical” actions $a$ are compared to find the respective minimum [19]. In the presented case, however, sample control actions are not necessary as the minimum of $l$ over $u$ is at $a=\bar{u}$ for all times.

For online adaptation, however, the parameters must be updated using only the current state and control obtained through (8). This can be achieved using (11) for $j=k$ with $x=x_{k}$ and $u=u_{k}$. In this case, observe that the updated parameter correlates to the coefficient used in the subsequent time step in (8) , i. e., $w_{k}=c_{k+1}$, $\forall k\in\mathbb{N}_{0}$.

Remark 3

There exists approaches which handle the issue of providing samples online and utilize parameters $w\in\mathbb{R}^{p}$, $p>1$. In [11], the use of recursive least-squares is suggested, updating the parameters each $p$ time steps, using the past $p$ state-action values. Provided, such “buffer” can be used, the parameters can be updated by sampling a minibatch uniformly from that buffer [12].

Remark 4

One should be aware of the distinct $TD(\lambda)$ [20, 3] approaches, representing a relevance weighting $0\leq\lambda\leq 1$ of recent temporal differences to earlier ones [18].

In the case $p=1$, the analogue online update rule to (11) for any of the $N$ coefficients may then read as

[TABLE]

$i=0,\dotsc,N-1$. It needs to be pointed out that different update rules may be used for the individual $c_{k}(i)$. Note that the coefficient is pulled out of the minimization objective in the last term on the right-hand side in (12) as it does not change the minimum (which is also true for (11)). The infimum is used since the feasible set is open.

As the stability constraint (10) must be satisfied, certain modifications to the respective update rule are necessary. For instance, in the optimization (12), which for all coefficients $C^{+}=[c_{k+1}(0),\dotsc,c_{k+1}(N-1)]^{\top}$ reads as

[TABLE]

with $\beta_{i}=l(x_{k}^{\ast}(i),u_{k}^{\ast}(i))+c_{k}(i)l(f(x_{k}^{\ast}(i),u_{k}^{\ast}(i)),\bar{u})$, the set (10) serves as optimization constraint.

For $k=0$, an initial guess of coefficients $c_{0}(i)$, $i=0,\dotsc,N-1$, is required, which is a crucial step in the sense that it may influence the performance severely. Unfortunately, no rule for choosing $c_{0}$ can be derived to this point (see also Section III-B).

Summarizing, the algorithm is depicted in Table I.

Allocation. As an alternative to an additional online optimization as in (13), the coefficients may be specified as the following: Looking at the set $\mathcal{W}_{k}$, one can choose

[TABLE]

Then, to ensure the inequality (10) in step $k$,

[TABLE]

which, under Assumption 2, may be restated as the specific allocation

[TABLE]

Proposition 3

Let Assumption 2 hold. If the initial coefficients are $c_{0}(i)=1$, for all $i\in\{0,\dotsc,N-1\}$, then the following bound regarding the decay rate (9) holds:

[TABLE]

Proof:

The result follows directly from (14) and (16). ∎

Observe that for $F$ fulfilling Assumption 1, smaller decay rates can be obtained in (9) than under Assumption 2 due to the different local decay rates of $F$. It should be noted, however, that the above allocation does not represent any learning strategy but mainly addresses the issue of maintaining stability while providing a specific coefficient upper bound which can be used in Proposition 1.

III Discussion

In the following, the temporal difference update is examined under the assumption that the stability related constraint (10) is satisfied while furthermore the initialization of the scheme is discussed.

III-A Online Adaptation

To elaborate on the meaning of the update (12) for the individual coefficients, it is assumed that, under (12), the stability constraint is satisfied for all times:

Assumption 3

For any $k\in\mathbb{N}_{0}$, under the sequences $\{x_{k}^{\ast}(i)\}_{i=0}^{N}$ and $\{u_{k}^{\ast}(i)\}_{i=0}^{N-1}$ from (8), the coefficients $c_{k+1}(i)$, $i=0,\dotsc,N-1$, obtained through (12) lie in $\mathcal{W}_{k}$.

This is purely to demonstrate certain properties of the update and does not affect the functionality of the algorithm. As mentioned previously, it is desirable to have $(\Delta l)_{k}$ in (6) as large as possible. Observe the following:

Proposition 4

Let Assumption 1 and 3 hold. Let $x_{k}^{\ast}(i)$ and $u_{k}^{\ast}(i)$, $i=0,\dotsc,N-1$ be the solution to (8) for any $k\in\mathbb{N}_{0}$. If, for any $k\in\mathbb{N}_{0}$, $c_{k}(i)\in(0,1]$ and $(x_{k}^{\ast}(i),u_{k}^{\ast}(i))\neq(\bar{x},\bar{u})$, then

[TABLE]

under (12), for any $i\in\{0,\dotsc,N-1\}$.

Proof:

If $c_{k}(i)\in(0,1]$ and $(x_{k}^{\ast}(i),u_{k}^{\ast}(i))\neq(\bar{x},\bar{u})$, then (12) yields

[TABLE]

where $\min_{a\in\mathbb{U}}\;l(f(x_{k}^{\ast}(i),u_{k}^{\ast}(i)),a)=l(f(x_{k}^{\ast}(i),u_{k}^{\ast}(i)),\bar{u})$. Since $(x_{k}^{\ast}(i),u_{k}^{\ast}(i))\neq(\bar{x},\bar{u})$, the last term on the right-hand side is strictly positive, which implies that $c_{k}(i)\in(0,1]\,\Rightarrow\,c_{k+1}(i)>1$, and subsequently (18). ∎

It can be deduced that whenever the algorithm is initialized with value $c_{0}(i)\in(0,1]$, $i=0,\dotsc,N-1$, all subsequent coefficients are $c_{k}(i)>1$, $k\in\mathbb{N}$, if state and control satisfy $(x_{k}^{\ast}(i),u_{k}^{\ast}(i))\neq(\bar{x},\bar{u})$.

Remark 5

By the above property of the coefficients tending toward higher magnitude and by definition of the Q-function being the infinite sum of stage cost, it is legitimate to directly impose a lower bound constraint $c_{k}(i)\geq 1$, $i\in\{0,\dotsc,N-1\}$, for any $k\in\mathbb{N}$, on (13).

Observe, when $(x_{k}^{\ast}(i),u_{k}^{\ast}(i))=(\bar{x},\bar{u})$, any $c_{k}(i)\in\mathbb{R}_{>0}$ is a feasible candidate to (12).

Recalling from Proposition 1 that there exists an upper bound on the (first) parameter $c_{k}(0)\leq\bar{c}$, the following assumption regarding (19) is made:

Assumption 4

For any sequences $\{x_{k}\}_{k=0}^{\infty}$ $\{u_{k}\}_{k=0}^{\infty}$, with $x_{k}\in\mathbb{X}$, $u_{k}\in\mathbb{U}$ and $(x_{k},u_{k})\rightarrow(\bar{x},\bar{u})$ for $k\rightarrow\infty$,

[TABLE]

The above Assumption 4 can be verified only using properties of the dynamic $f$ and the stage cost $l$.

Remark 6

Assumption 4 does not influence the functionality of the proposed algorithm but only provides a bound $\bar{c}$ regarding the suboptimality estimate established in Proposition 1. Furthermore, referring to (12) and (13) – for bounded, nonzero $(x_{k}^{\ast}(i),u_{k}^{\ast}(i))$ and bounded $0<c_{k}(i)<\infty$ – an infimum is attained for bounded $c_{k+1}(i)<\infty$.

Hence, different upper bounds $\bar{c}$ are obtained by Proposition 3 and by the supremum of the right-hand side of Eq. (19) according to the value in Assumption 4. This implies that different performances may be achieved with respect to the estimate (6), specifically regarding the difference between $\delta_{0}$ and the sum over $(\Delta l)_{k}$.

III-B Initialization

Notice, that in (6), $W(x_{k})$ relates to $\tilde{V}^{\infty}(x_{k}^{\ast},k)$ whereas the applied control is given by $\mu(x_{k})=u_{k}^{\ast}(0)$. Since, if Assumption 2 holds, $V^{\infty}(x_{0})\leq\tilde{V}_{\text{MPC}}^{\infty}(x_{0})$ for any $x_{0}\in\mathbb{X}$, inequality (6) can be extended to give an indirect comparison with the MPC cost. Denoting $\gamma_{0}:=\tilde{V}_{\text{MPC}}^{\infty}(x_{0})-V^{\infty}(x_{0})$, it follows that under Assumption 2, $\gamma_{0}\geq 0$ and furthermore

[TABLE]

where $x_{0}^{\ast}=x_{0}\in\mathbb{X}$. This implies that if $\gamma_{0}$ is large enough and the infinite sum over $(\Delta l)_{k}$ has a sufficiently high (finite) value, $\Delta V_{N}$ can be made non-positive and thus

[TABLE]

This comparison, however, is in so far indirect as it does not involve the actual infinite sum of stage costs under the control actions generated through (3) (with Assumption 2). Observe that this sum, by optimality, must be larger than $V^{\infty}$ from (2) and thus from (6),

[TABLE]

with $\{x_{k}^{\text{MPC}}\}_{k=0}^{\infty}$ and $\{u_{k}^{\text{MPC}}(0)\}_{k=0}^{\infty}\in\mathbb{U}_{\text{ad}}$ the state and control trajectory, respectively, under (3). Here, only limited information about the relationship between $V^{\infty}$ and the first term on the right-hand side of (22) is available though.

Due to (in general) missing information about the solution to the IHOCP (2), no specific allocation rule for the coefficients in (8) at $k=0$ can be given. Therefore, starting the scheme (8) equivalently to MPC (3) , i. e., $c_{0}(i)=1$, $i=0,\dotsc,N-1$, is reasonable. Yet, as elaborated in Proposition 4, the coefficients may tend to a higher value, which suggests an initialization $c_{0}(i)>1$.

IV Simulation Study

Next, the time-variant stage cost approach (8) is compared to the standard MPC scheme (3) using two test systems.

First, a linear mass spring damper system

[TABLE]

with specific $m=1\,\,\text{Kg}$, $d=0.2\,\,\frac{\text{Ns}}{\text{m}}$ and $s=1\,\,\frac{\text{N}}{\text{m}}$ is used under Euler discretization with sampling time $\Delta t=0.7$ (exact discretization is not used for consistency with the second example). The stage cost and the horizon are set to $l(x,u)=\|x\|^{2}+u^{2}$ and $N=10$, respectively.

Secondly, the nonlinear system from [15]

[TABLE]

is Euler discretized using $\Delta t=0.1$. The stage cost and the horizon are set to $l(x,u)=x_{2}^{2}+u^{2}$ and $N=5$, respectively.

For linear and nonlinear system control, initial sets of coefficients $c_{0}(i)=5$, $i\in\{0,\dotsc,9\}$ and $c_{0}(i)=20$, $i\in\{0,\dotsc,4\}$, are chosen assuming no further knowledge about the optimal value function. Under variation of the initial discrepancies $-\gamma_{0}+\delta_{0}$ in (21) by using other initial coefficients, the obtained simulation results may be different. The terminal costs (and a corresponding controller) are obtained by finding a local LQ control and by using the optimal (continuous time) value function given in [15], respectively, for the two systems. The terminal set was obtained using level sets $\mathbb{X}_{f}:=\{x\in\mathbb{X}:\,F(x)\leq 0.1\}$. Regarding (21), a comparison of MPC (3) and the proposed scheme (8) can be seen in Fig. 1 as well as in Fig. 2 for either system. The scale on the right side of the contour depicts the difference between the (quasi-) infinite sum of stage costs under either algorithm: positive values denote superiority of MPC and negative values indicate better performance under (8), using the presented initial coefficients. Though the performances are dependent on a, roughly speaking, suitable initialization of the coefficients, they may also vary with the specific coefficient update rule used , e. g., either by optimization or allocation. In the simulation, the algorithm presented in Table I is used. Additionally, for the nonlinear system, a lower bound on $c_{k}(i)\geq 1$, for all $i\in\{0,\dotsc,4\}$, was imposed on the optimization in (13) according to Remark 5.

It can be seen in the proposed approach may outperform MPC for certain initial states with the given configuration, whereas elsewhere MPC appears superior. As pointed out in Sec. III-B, the initial coefficients are, to this point, chosen arbitrarily, the performance under (8) is not guaranteed to be better with respect to the IHOCP. However, the results indicate the potential of cost reduction when the stage cost is weighted in the online optimization. This may be of special interest for short predictive horizon problems, as then the computational load is within limits (also regarding the additional optimization of the coefficients).

V Conclusion

In this work, the rather general question of MPC performance with respect to a nonlinear IHOCP problem is addressed. In the framework of stabilizing MPC, specifically that using a stabilizing terminal set constraint and terminal cost, some properties of a modified finite-horizon cost function with corresponding decay are presented and put in relation with the solution to the IHOCP. It is argued that an approximation of the IHOCP by the proposed finite-horizon scheme can induce a higher decay rate which could subsequently reduce the “degree of suboptimality”. Through introduction of coefficients to the standard MPC problem, this decay rate can be achieved via an additional optimization problem while specific allocation can also be used to circumvent online optimization and guarantee stability.