Practical Reinforcement Learning of Stabilizing Economic MPC

TL;DR

This paper introduces an improved reinforcement learning algorithm tailored for stabilizing economic model predictive control, effectively combining RL and MPC to handle nonlinear dynamics and constraints without requiring precise models.

Contribution

The paper presents a novel RL algorithm specifically designed for MPC, enhancing stability and performance in control tasks with uncertain models.

Findings

The proposed method successfully stabilizes complex control systems in simulations.

It outperforms traditional MPC in scenarios with model inaccuracies.

The approach demonstrates robustness and improved closed-loop performance.

Abstract

Reinforcement Learning (RL) has demonstrated a huge potential in learning optimal policies without any prior knowledge of the process to be controlled. Model Predictive Control (MPC) is a popular control technique which is able to deal with nonlinear dynamics and state and input constraints. The main drawback of MPC is the need of identifying an accurate model, which in many cases cannot be easily obtained. Because of model inaccuracy, MPC can fail at delivering satisfactory closed-loop performance. Using RL to tune the MPC formulation or, conversely, using MPC as a function approximator in RL allows one to combine the advantages of the two techniques. This approach has important advantages, but it requires an adaptation of the existing algorithms. We therefore propose an improved RL algorithm for MPC and test it in simulations on a rather challenging example.

Figures

Equations

$$Q_{\star }\left(s,a\right)$$

$$V_{\star }\left(s\right)$$

$$\delta$$

$$\theta$$

$$\displaystyle Q^{N}_{\theta}(s,a)=\min_{z}\$$

$$+\sum _{k=0}^N{\gamma }^k\left(s_k^{\top }{W}_s{s}_k+w_s^{\top }{s}_k\right)$$

$$\displaystyle\mathrm{s.t.}\$$

$$x_{k+1}=f_{\theta }\left(x_k,u_k\right),$$

$$g_{\theta }\left(u_k\right)\le 0,$$

$$h_{\theta }\left(x_k,u_k\right)\le s_k,h_{\theta }^{\mathrm{f}}(x_N)\le s_N,$$

$${\pi }_{\theta }^N(s)=\arg \underset{a}{\min }{Q}_{\theta }^N(s,a),V_{\theta }^N(s)=\underset{a}{\min }{Q}_{\theta }^N(s,a).$$

$${\hat{V}}^N(s)=\underset{\pi }{\min }\mathbb{E}\left[{\gamma }^N{V}_{\star }({\hat{s}}_N^{{\hat{\pi }}^N})+\sum _{k=0}^{N-1}{\gamma }^k\hat{\ell }({\hat{s}}_k^{{\hat{\pi }}^N},\hat{\pi }({\hat{s}}_k^{{\hat{\pi }}^N}))\right]$$

$$\left|\mathbb{E}\left[V_{\star }\left({\hat{s}}_k^{{\pi }_{\star }}\right)\right]\right|<\infty ,\forall s\in \mathcal{S}.$$

$${\delta }_k=\ell (s_k,u_k)+\gamma \mathbb{E}\left[V_{\hat{\theta }}^N(s_{k+1})\right]-Q_{\hat{\theta }}^N(s_k,a_k)=0.$$

$$\delta =\ell (s,a)+\gamma V_{\theta }^{N-1}(f(s,a))-Q_{\theta }^N(s,a).$$

$$Q_{\theta }^1(s,a)$$

$$V_{\theta }^0(s)$$

$$\displaystyle\delta=\gamma\left[\begin{array}[]{@{}c@{}}s\\ a\end{array}\right]^{\top}\left(\left[\begin{array}[]{@{}cc@{}}A&B\end{array}\right]^{\top}\hat{P}\left[\begin{array}[]{@{}cc@{}}A&B\end{array}\right]-\left[\begin{array}[]{@{}cc@{}}\hat{A}&\hat{B}\end{array}\right]^{\top}\hat{P}\left[\begin{array}[]{@{}cc@{}}\hat{A}&\hat{B}\end{array}\right]\right)\left[\begin{array}[]{@{}c@{}}s\\ a\end{array}\right].$$

$$\hat{K}=(R+\gamma {\hat{B}}^{\top }\hat{P}\hat{B}{)}^{-1}(S^{\top }+{\hat{B}}^{\top }\hat{P}\hat{A})$$

$$V_{\theta }^1(s)$$

$$=s^{\top }(T+\gamma {\hat{A}}^{\top }\hat{P}\hat{A}-(S+\gamma {\hat{A}}^{\top }\hat{P}\hat{B})\hat{K})s,$$

$$\displaystyle Q_{\theta}^{N}(s,a)=\min_{z}\$$

$$\displaystyle\mathrm{s.t.}\$$

$$Cz\le d,$$

$$\displaystyle Q_{\theta}^{1}(s,a)=\left[\begin{array}[]{@{}c@{}}s\\ a\end{array}\right]^{\top}\left[\begin{array}[]{@{}ll@{}}\bar{T}&\bar{S}\\ \bar{S}^{\top}&\bar{R}\end{array}\right]\left[\begin{array}[]{@{}c@{}}s\\ a\end{array}\right],$$

$$Q_{\theta }(s,a)=\sum _{\bar{s}\in \mathcal{S},\bar{a}\in \mathcal{A}}{\theta }_{\bar{s},\bar{a}}{\varphi }_{\bar{s},\bar{a}}(s,a),$$

$$\sum _{\bar{s}\in \mathcal{S},\bar{a}\in \mathcal{A}}{\theta }_{\bar{s},\bar{a}}{\delta }_{\bar{s},\bar{a}}(s,a)=\alpha \delta .$$

$$\Delta {\theta }^{\ast }$$

$$\displaystyle\min_{\theta}\$$

$$\displaystyle\mathrm{s.t.}\$$

$$\displaystyle\mathcal{L}_{\theta}^{N}(s,y)=\$$

$$+\sum _{k=0}^{N-1}{\chi }_{k+1}^{\top }\left(f_{\theta }\left(x_k,u_k\right)-x_{k+1}\right)+{\nu }_k^{\top }{g}_{\theta }\left(u_k\right)$$

$$+{\gamma }^k{\ell }_{\theta }(x_k,u_k)+{\mu }_k^{\top }{h}_{\theta }\left(x_k,u_k\right)+{\zeta }^{\top }(u_0-a),$$

$${\nabla }_{\theta }{Q}_{\theta }^N(s,a)={\nabla }_{\theta }{\mathcal{L}}_{\theta }^N(s,y^{\star })$$

$${\nabla }_{\theta }\underset{a}{\min }{Q}_{\theta }^N(s,a)={\nabla }_{\theta }{\mathcal{L}}^N(s,y^{\diamond }),$$

$${\nabla }_{\theta }{y}^{\star }=-{\nabla }_{\theta }{\xi }_{\theta }^N(s,y^{\star }){\nabla }_y{\xi }_{\theta }^N(s,y^{\star }{)}^{-1},$$

$$\theta =(H_{\lambda },h_{\lambda },c_{\lambda },H_{V^{\mathrm{f}}},h_{V^{\mathrm{f}}},c_{V^{\mathrm{f}}},H_l,h_l,c_l,c_f,x_{\mathrm{l}},x_{\mathrm{u}}).$$

Full text

I Introduction

Reinforcement learning (RL) is a model-free control technique which recursively updates the controller parameters in order to achieve optimality. Once the controller parameters have been learned, the controller will implement the control action which yields an infinite-horizon optimal cost for each initial condition [1]. RL has drawn increasing attention thanks to the striking results obtained in beating chess and go masters [2], and in learning how to make a robot walk or fly without supervision [3, 4].

In order to be able to solve the RL problem in practice, function approximation strategies must be employed. Such approximations typically rely on a set of basis functions, or features, and corresponding parameters that multiply them. In this case, the function approximator is linear in the parameters. However, nonlinear function approximators are commonly deployed using Deep Neural Networks (DNN).

Model Predictive Control (MPC) is a model-based technique which exploits a model of the system dynamics to predict the system’s future behavior and optimize a given performance index, possibly subject to input and state constraints [5, 6, 7]. The success of MPC is due to its ability to enforce constraints and yield optimal trajectories. While a plethora of efficient algorithms for the online solution of MPC problems has been developed, the main drawback of this control technique is the need of identifying the open-loop model offline, which is typically the most time-consuming phase of control design.

The advantages of MPC and RL can be combined together by framing MPC as a function approximator within an RL context. The tuning parameters of the MPC problem (e.g., cost weighting matrices, model parameters, etc.) can be framed as parameters of a nonlinear function approximator, namely the MPC optimization problem. This setup can be adopted to approximate the feedback policy, the value function, or the action-value function.

The main advantages of combining MPC with RL are (a) the ease of introducing a constraint-enforcing policy within RL, (b) the possibility of improving existing models and controllers by using them as an initial guess in RL, and (c) the possibility of interpreting the learned algorithm in a model-based framework.

The combination of learning and control techniques has been proposed in, e.g., [8, 9, 10, 11]. Some attempts at combining RL and the linear quadratic regulator have been presented in [12, 13]. To the best of our knowledge, however, [14] is the first work proposing to use NMPC as a function approximator in RL.

In this paper, building on the ideas from [14], we further analyse the combination of RL and MPC. The main contribution of this paper is the development of an improved algorithmic framework tailored to the problem formulation which overcomes some shortcomings of basic RL algorithms. We demonstrate the potential of our approach in simulations on an involved nonlinear example from the process industry.

This paper is structured as follows. Section II introduces $Q$-learning. The main contributions of this paper are presented in Section III, which discusses the use of MPC as a function approximator in RL, and Section IV, which presents an algorithm adaptation, tailored to the problem. Some numerical examples are given in Section V. The paper is concluded by Section VI which also outlines future research directions.

II Reinforcement Learning

Consider a Markov Decision Process (MDP) with state transition dynamics $\mathbb{P}[s_{+}|s,a]$, where $s$ and $a$ denote the states and actions (or controls) respectively. We also introduce the stage cost $\ell(s,a)$ and the discount factor $0<\gamma\leq 1$. The action-value function $Q_{\star}(s,a)$ and value function $V_{\star}(s)$ associated with the optimal policy $\pi_{\star}\left(s\right)$ are defined by the Bellman equations:

[TABLE]

$Q$-learning parametrizes the action value function as $Q_{\theta}(s,a)$, where $\theta$ is a vector of parameters whose values have to be learned, and aims at minimizing $\|Q_{\star}(s,a)-Q_{\theta}(s,a)\|_{2}^{2}$. Given a state-action pair $(s,a)$ and the next state $s_{+}$, standard algorithms [1] update the parameter using

[TABLE]

where $\delta$ is known as the Temporal-Difference (TD) error. In batch policy updates $\tilde{\theta}$ is kept constant for $N_{\mathrm{upd}}$ steps after which it is updated as $\tilde{\theta}\leftarrow\theta$. In instantaneous policy updates $N_{\mathrm{upd}}=1$ such that $\tilde{\theta}$ is updated at every time instant.

It is important to underline that, while in this paper we focus on $Q$-learning (which has been successfully deployed on some applications, e.g., [15, 16, 17]), our approach can be readily deployed on other TD-algorithms such as SARSA [18]. Even in approaches directly optimizing the policy, learning the action-value function is often necessary [19, 20], such that the developments of this paper apply to a fairly broad class of RL algorithms.

While a very common choice is to use Deep Neural Networks (DNN) as function approximators, it is hard to analyze closed-loop stability in DNN-based RL. Moreover, in case a controller is already available, information on how to control the system cannot be easily incorporated in the problem formulation. For this reason, we propose to use MPC as a function approximator, since it makes it possible to enforce closed-loop stability guarantees and an existing controller can be used as initial guess for the algorithm.

III MPC-Based RL

The use of MPC to parametrize the action-value function has been first advocated in [14]. In the following, we first present the function approximation used and recall the most important result in Theorem 1, then we further analyze the properties of MPC-based RL and in the next section we will propose a new variant of the $Q$-learning algorithm tailored to MPC-based RL.

III-A Parametrization of the Function Approximations

We parametrize the action-value function using an MPC scheme of the form

[TABLE]

where $z=(x_{0},u_{0},s_{0},\ldots,x_{N})$. Consequently, we obtain

[TABLE]

Note that the policy $\pi_{\theta}^{N}(s)$ and value function $V_{\theta}^{N}(s)$ are equivalently obtained by solving Problem (3) with constraint $u_{0}=a$ removed. In order to address feasibility issues in Problem (3), in the presence of state-dependent constraints we adopt an exact relaxation of such constraints [21].

Remark 1

MPC formulations in which the stage cost is not positive-definite are commonly referred to as Economic MPC. In order to enforce closed-loop stability and the existence of the MPC solution, we assume that $\ell_{\theta}$ and $V^{\mathrm{f}}_{\theta}$ are positive definite functions. Therefore, in order to deal with the situation in which the true stage cost $\ell$ is not positive-definite, we have introduced the arrival penalty $\lambda_{\theta}$. For all details on this topic, we refer to [14] and references therein.

Remark 2

The proposed setup readily accommodates for formulations in which the cost penalizes deviations from a given reference. In that case, the stage cost (both in RL and MPC) depends on a reference, passed to the problem as an exogenous signal. Input-output model formulations also readily fit in the proposed framework.

Among the desirable properties of the proposed formulation, we mention nominal stability guarantees, and the possibility to introduce constraints accounting for, e.g., actuator limitations, safe operation of the system, etc. While being aware that the current framework is not able to fully exploit these advantages, with the presented developments we aim at constructing a sound basis which will be used in future research with the intent of developing self-tuning, safe, and stable economic MPC controllers.

III-B Learning the Model: RL and System Identification

We recall the following fundamental theorem from [14], which states that the optimal value and action-value functions as well as the optimal policy can be learned even by using MPC based on a state transition model $\hat{f}$ which is different from the true model $f$. This also entails that there is no guarantee (and no need) that the RL algorithm will learn a physically meaningful model.

Theorem 1** ([14])**

*Consider a given (possibly stochastic) state transition model $\hat{s}_{k+1}=\hat{f}(\hat{s}_{k},\hat{a}_{k})$, possibly different from the true model $f(s_{k},a_{k})$. Define the optimal value function *

[TABLE]

associated with stage cost $\hat{\ell}$, and the terminal cost $V_{\star}(s)$ over an optimization horizon $N$. Define $\hat{s}^{\hat{\pi}}_{0,\ldots,N}$ as the (possibly stochastic) trajectories of the state transition model $\hat{f}$ under a policy $\hat{\pi}$, starting from $\hat{s}^{\hat{\pi}}_{0}=s$, and $\hat{\pi}^{N}$ the optimal policy associated with $\hat{V}_{N}(s)$ and $\hat{Q}^{N}(s,a)$ the associated action-value function. Consider the set $\mathcal{S}$ such that

[TABLE]

Then, $\exists\ \hat{\ell}$ such that the following identities hold on $\mathcal{S}$:

• (i)

$\hat{V}^{N}(s)=V_{\star}(s)$**

• (ii)

$\hat{\pi}^{N}\left(s\right)=\pi_{\star}\left(s\right)$**

• (iii)

$\hat{Q}_{N}\left(s,a\right)=Q_{\star}\left(s,a\right)$* for the inputs $a$ such that $\left|\,\mathbb{E}\left[V^{\star}\left(\hat{s}_{+}\right)\,|\,s,a\right]\,\right|<\infty$.*

We can now further clarify the theorem and formalize a form of “orthogonality” between reinforcement learning and system identification.

Corollary 2

Let us split the parameter as $\theta=(\theta_{f},\theta_{Q})$, such that the model $f_{\theta}$ only depends on $\theta_{f}$ and assume a perfect parametrization, such that, under adequate exploration of the state-action space, RL learns $Q$ and $V$ perfectly with $\hat{\theta}\neq\theta$, i.e.,

[TABLE]

Then, leaving the task of identifying $\theta_{f}$ to a separate identification algorithm is not detrimental to the learning of the optimal value, action-value function, and policy.

Unfortunately, Corollary 2 applies only to the case of perfect parametrization. In practice, function approximation with imperfect parametrization and lack of excitation can destroy this form of orthogonality. Ongoing research is currently further investigating such aspects.

III-C On the Parametrization of $V$ and $Q$

Since the Bellman principle of optimality states that $Q_{\theta}^{N}(s,a)=\ell_{\theta}(s,a)+V^{N-1}_{\theta}(f_{\theta}(s,a))$, one could be tempted to replace $V_{\theta}^{N}(s)=\min_{a}Q_{\theta}^{N}(s,a)$ by $V_{\theta}^{N-1}(s)$ in the computation of the TD error. We discuss in the following lemma why this approach can minimize the TD error while not learning the correct action-value function and, consequently, not delivering the optimal policy.

Lemma 3

Consider computing the TD error (2a) by replacing $\min_{a}Q_{\theta}^{N}(s,a)$ by $V_{\theta}^{N-1}(s)$ to obtain

[TABLE]

Then, it is possible to obtain $\delta=0$ without minimising the error $\|Q^{\star}(s,a)-Q_{\theta}^{N}(s,a)\|^{2}$ and, therefore, without learning the optimal policy.

Proof:

We prove the Lemma by a simple counter example. Consider the LQR case, with $\theta=(\hat{A},\hat{B},\hat{P})$ and

[TABLE]

Then, (4) reads as

[TABLE]

Assume now that the correct model is identified, i.e., $\hat{A}=A$, $\hat{B}=B$. Then, the TD error is independent of $\hat{P}$, which can be chosen as desired. This implies that, even though we use a perfect parametrization of the action-value function, $Q_{\theta}^{1}(s,a)\neq Q_{\star}(s,a)$. Consequently, the learned feedback given by

[TABLE]

is not uniquely defined. ∎

For comparison, we consider now the standard case, in which the TD error is computed using

[TABLE]

where $\hat{K}$ is given by (5). In this case, if $\hat{A}=A$, $\hat{B}=B$, $\hat{P}$ must solve the algebraic Riccati equation associated with the given stage cost and model. It is important to stress, however, that infinitely many solutions exist in general such that the true model will not be identified. As an example we provide $\theta_{1}=(0.5\hat{A},0.5\hat{B},4\hat{P})$: it is immediate to verify that $Q_{\theta_{1}}^{1}(s,a)\equiv Q_{\theta}^{1}(s,a)$.

III-D Condensed Model-Free Parametrization

Motivated by the previous considerations on the possibility of learning a wrong model, we propose next a formulation which is truly model-free as it does not include model parameters as coefficients to be learned. For simplicity we only focus here on the case of linear dynamics with a quadratic cost. We define $z=(x_{0},u_{0},\ldots,u_{N-1})$; then

[TABLE]

where $\theta=(M,C,d,m)$ and $M$ is a symmetric (typically dense) matrix. Problem formulation (6) is sometimes used in MPC and optimal control, in which case the parameters $\theta$ are related to the system dynamics and cost [22]. In this case, the dependence of $V$ and $Q$ on the parameters is less nonlinear than in the model-based parametrization, see (3). However, the introduction of a prediction horizon $N>1$ possibly introduces more parameters than the standard formulation. A special case occurs when there are only input constraints and they are known: in this case $C$ can be fixed and only $M$, $m$ need to be learned. The LQR case simplifies to

[TABLE]

such that $K=\bar{R}^{-1}\bar{S}^{\top}$ and $\pi^{\star}(s)=-Ks$

IV Algorithm

The main practical difficulties related to using MPC as a function approximator in RL are that: (a) the function approximator is nonlinear in the parameters $\theta$, and (b) the MPC problem is guaranteed to have a meaningful solution only if the cost is positive-definite. In this section we propose an algorithm that addresses both issues.

We start by recalling that the main motivation for the stochastic gradient approach typically used in $Q$-learning stems from an equivalence with the table-lookup case, i.e., when the state-action space is discrete and the action-value function is parametrized as

[TABLE]

and $\phi_{\bar{s},\bar{a}}(s,a)=1$ if $\bar{s}=s,\bar{a}=a$ and [math] otherwise. In this case, $Q_{\theta}$ is linear in the parameter $\theta$. Parameter $\alpha$ in (2b) is used in order to approximate the expected value of the TD error: one can roughly interpret it as the inverse of the amount of samples over which the average is computed.

In this case, the update (2b) obtained with the (exact) function approximation matches that of the enumeration

[TABLE]

The update (2b) can also be written as $\theta\leftarrow\theta+\alpha\Delta\theta^{*}$ with

[TABLE]

We remark that this is not the case if $\phi$ is not normalized or if $Q$ is a nonlinear function of the parameter $\theta$. This also implies that parameter $\alpha$ loses its original meaning, as it is also used to dampen the stochastic gradient step in order to enforce convergence.

We propose to apply the update $\theta\leftarrow\theta+\alpha(\theta^{*}-\theta)$ where $\theta^{*}$ is the solution of

[TABLE]

Note that we solve Problem (14) to full convergence at each step. In case of a linear parametrization of $Q_{\theta}$, convergence is obtained in one step by using a Gauss-Newton Hessian approximation, in case Constraints (14b) are inactive. This also directly solves the issue of scaling, present in the table-lookup case if $\phi$ is not normalized. Finally, if instead of solving Problem (14) to full convergence constraints (14b) are neglected and one takes a single full Newton step, the update (2b) is recovered.

We highlight next the main features of Formulation (14):

• •

Globalization: globalization strategies such as line search ensure descent and, therefore, convergence. Descent is typically not enforced in stochastic gradient approaches such as (2), which could take a step which yields a larger TD error than with the previous parameter estimate.

• •

Positive-definiteness enforcement: Constraints (14b) guarantee that the cost is positive-definite and, therefore, let the MPC problem be stabilizing and well-posed. Without this constraint, even with a well-posed, positive-definite cost, throughout the RL iterates one might obtain an indefinite cost yielding an unbounded solution. We have written the positive-definiteness Constraints (14b) using the Hessian of $\ell$ and $V^{\mathrm{f}}$, which is reasonable if both functions are quadratic. However, other approaches relying on richer function approximations are possible, e.g. using sum-of-squares techniques.

• •

Best fit: by solving Problem (14) to full convergence, the step $\theta^{*}-\theta$ is the one which minimizes $\delta$, analogously to the lookup-table case. Therefore, the choice of parameter $\alpha$ can be done purely by considerations about the expected value approximation.

In summary, the main differences with the standard update (2b) are: (a) positive-definiteness enforcement, (b) insensitivity to parameter scaling and (c) guarantee of improvement at each step through globalization and full convergence.

IV-A Derivative Computation

We detail next how to compute the derivatives of the action-value function with respect to the parameters. To this end, we define the Lagrangian function underlying NMPC problem (3) as

[TABLE]

where $\chi,\mu,\nu,\zeta$ are the multipliers associated with constraints (3b)-(3e) and $y=(z,\chi,\mu,\nu,\zeta)$. Note that, for $\zeta=0$, $\mathcal{L}_{\theta}^{N}(s,y)$ is the Lagrangian function associated to the NMPC problem defining the value function $\min_{a}Q_{\theta}(s,a)$. We observe that [23]

[TABLE]

holds for $y^{\star}$ given by the primal-dual solution of (3). Note that this equality holds because constraints (3b) are not an explicit function of $\theta$. The gradient (16) is therefore straightforward to build as a by-product of solving the NMPC problem (3). We additionally observe that

[TABLE]

where $y^{\diamond}$ is given by the primal-dual solution to (3) with constraint $u_{0}=a$ removed and $\zeta^{\diamond}=0$.

We remark that second-order derivatives can also be computed, though in general they depend on the derivative of the optimal primal-dual solution with respect to the parameters, given by

[TABLE]

where $\xi_{\theta}^{N}(s,y)$ gathers the primal-dual KKT conditions underlying the NMPC scheme (3). For a complete discussion on parametric sensitivity analysis of NLPs we refer to [23] and references therein.

V Numerical Example

We consider an example from the process industry, i.e. the evaporation process modelled in [24, 25] and used in [26, 27] to demonstrate the potential of economic MPC in the nominal case. For the sake of brevity we omit the model equations and non-quadratic economic cost function, which include states $(X_{2},P_{2})$ (concentration and pressure) and controls $(P_{100},F_{200})$ (pressure and flow). All details can be found in [26, 27]. The model further depends on concentration $X_{1}$, flow $F_{2}$, and temperatures $T_{1},T_{200}$, which are assumed to be constant in the control model. In reality, these quantities are stochastic with variance $\sigma_{X_{1}}=1$, $\sigma_{F_{1}}=2$, $\sigma_{T_{1}}=8$, $\sigma_{T_{200}}=5$, and mean centred on the nominal value. Bounds $(25,40)\leq(X_{2},P_{2})\leq(100,80)$ on the states and $100\leq(P_{100},F_{200})\leq 400$ on the controls are present. In particular, the bound $X_{2}\geq 25$ is introduced in order to ensure sufficient quality of the product. All state bounds are relaxed as in (3e).

We parametrize an NMPC controller as in (3), i.e., a nonlinear non-condensed MPC formulation, with $N=10$, $\lambda_{\theta},V^{\mathrm{f}}_{\theta},l_{\theta}$ quadratic functions defined by Hessian $H_{\dagger}$, gradient $h_{\dagger}$, and constant $c_{\dagger}$, $\dagger=\{\lambda,V^{\mathrm{f}},l\}$. The model is parametrized as the nominal model with the addition of a constant, i.e., $f_{\theta}(x,u)=f(x,u)+c_{f}$. The control constraints are fixed and the state constraints are parametrized as simple bounds, i.e., $h_{\theta}(x,u)=(x-x_{\mathrm{l}},x_{\mathrm{u}}-x)$. The vector of parameter therefore reads as:

[TABLE]

Constants $w_{s}=1$, $W_{s}=I$ are fixed and assumed to reflect the known cost of violating the state constraints.

We use the batch policy update with $\alpha=10^{-2}$ and update the parameters with the learned ones every $N_{\mathrm{upd}}=500$ time steps. In order to induce enough exploration, we use an $\epsilon$-greedy policy which is greedy $90\,\%$ of the samples, while in the remaining $10\,\%$ we apply the action

[TABLE]

where $\mathrm{sat}(\cdot,u_{\mathrm{l}},u_{\mathrm{u}})$ saturates the input between its lower and upper bounds $u_{\mathrm{l}},u_{\mathrm{u}}$, respectively.

We initialize the ENMPC scheme by the naive initial guess $H_{l}=I$, $x_{\mathrm{l}}=(25,40)$, $x_{\mathrm{u}}=(100,80)$, while all other parameters are [math]. While at every step we do check that $H_{l},H_{V^{\mathrm{f}}}\succ 0$, during the learning phase the parameters never violate this constraint. As displayed in Figure 1, the algorithm converges to a constant parameter value while reducing the average TD-error. If the standard parameter update (2b) is applied, RL does not converge with the proposed $\alpha$. If a smaller value is used, the algorithm does not diverge but the parameters $\theta$ are updated very slowly, making the approach impractical.

We performed a simulation to compare the RL-tuned NMPC scheme to the naive initial guess and the NMPC tuned using the economic-based approach proposed in [27], which relies on the nominal model. The economic gain obtained by RL is approximately $14\,\%$ and $12\,\%$ respectively. The effectiveness of the economically-tuned NMPC had been demonstrated in [27] in the absence of stochastic perturbations. In the considered scenario we observe that, while the economically-tuned scheme still performs better than the naively-tuned one, the RL tuning is able to significantly outperform the two other NMPC schemes by explicitly accounting for the stochastic perturbations.

The concentration $X_{2}$ and the difference in instantaneous cost between the RL-tuned and the naively-tuned scheme are displayed in Figure 2. In particular, one can see that RL is trying to stabilize the concentration $X_{2}$ to a value which is higher than the nominally optimal one: the optimum in the presence of perturbations is obtained as a compromise between the loss due to operating at $X_{2}>25$ and the cost of violating the constraint $X_{2}\geq 25$. Indeed, the constraint is violated but only rarely and by small amounts.

VI Conclusions and Outlook

In this paper we analyzed the use of reinforcement learning to tune MPC schemes, aiming at a self-tuning controller which guarantees stability, safety (i.e., constraint satisfaction), and optimality. In order to be able to apply this approach in practice, we have proposed an improved algorithm based on $Q$-learning and we have tested it in simulations.

Future work will consider several research directions including: (a) further improvements in the algorithmic framework with the aim of developing data-efficient algorithms; (b) developing new algorithms for other RL paradigms, such as policy gradient methods; (c) further investigating the combination of system identification and RL in order to best update the MPC parameters while guaranteeing safety.