Optimistic planning for the near-optimal control of nonlinear switched discrete-time systems with stability guarantees

TL;DR

This paper introduces OPmin, an improved optimistic planning algorithm for nonlinear switched systems, addressing previous limitations and providing near-optimal, stable control with broader cost function applicability.

Contribution

The paper develops OPmin, a modified optimistic planning algorithm that overcomes key limitations of OP, enabling near-optimal and stable control for nonlinear switched systems with general cost functions.

Findings

OPmin achieves near-optimal control guarantees.

OPmin ensures stability under certain assumptions.

The algorithm broadens applicability to various cost functions.

Abstract

Originating in the artificial intelligence literature, optimistic planning (OP) is an algorithm that generates near-optimal control inputs for generic nonlinear discrete-time systems whose input set is finite. This technique is therefore relevant for the near-optimal control of nonlinear switched systems, for which the switching signal is the control. However, OP exhibits several limitations, which prevent its application in a standard control context. First, it requires the stage cost to take values in [0,1], an unnatural prerequisite as it excludes, for instance, quadratic stage costs. Second, it requires the cost function to be discounted. Third, it applies for reward maximization, and not cost minimization. In this paper, we modify OP to overcome these limitations, and we call the new algorithm OPmin. We then make stabilizability and detectability assumptions, under which we derive near-optimality guarantees for OPmin and we show that the obtained bound has major advantages compared to the bound originally given by OP. In addition, we prove that a system whose inputs are generated by OPmin in a receding-horizon fashion exhibits stability properties. As a result, OPmin provides a new tool for the near-optimal, stable control of nonlinear switched discrete-time systems for generic cost functions.

Tables

Table 1. Table I: Estimated running cost for various budgets and initial conditions.

			Budget
		30	300	3000
	${[10,15]}^{\top }$	199015	13757	12609
Initial States	${[-1,1.5]}^{\top }$	314	28	22
	${[-15,-10]}^{\top }$	128184477	46875	42952
	${[10,-15]}^{\top }$	14180	2802	2615

Equations

$$x_{k+1}=f_{u_k}(x_k),$$

$$J_{\gamma ,\infty }(x,\mathbf{u}):=\sum _{k=0}^{\infty }{\gamma }^k{\ell }_{u_k}(\varphi (k,x,\mathbf{u}{|}_k)),$$

$$J_{\gamma ,d(x)}(x,\mathbf{u}):=\sum _{k=0}^{d(x)}{\gamma }^k{\ell }_{u_k}(\varphi (k,x,\mathbf{u}{|}_k)),$$

$$V_{\gamma ,d(x)}(x):=\underset{\mathbf{u}}{\min }{J}_{\gamma ,d(x)}(x,\mathbf{u}),$$

$$V_{\gamma ,\infty }(x)\le {\overline{\alpha }}_V(\sigma (x)).$$

$$W(x)$$

$$W(f_u(x))-W(x)$$

$$V_{\gamma ,d(x)}(x)\le V_{\gamma ,\infty }(x)\le V_{\gamma ,d(x)}(x)+v_{\gamma ,d(x)}(x),$$

$$V_{\gamma ,d}(x)\le V_{\gamma ,\infty }(x)\le V_{\gamma ,d}(x)+{\hat{v}}_{d(x)}(x)$$

$$x_{k+1}\in f_{{\mathcal{U}}_{\gamma ,d(x_k)}^{\ast }(x_k)}(x_k)=:F_{\gamma ,d(x_k)}^{\ast }(x_k),$$

$$\sigma (\varphi (k,x))\le \max \{\beta (\sigma (x),k),\delta \}.$$

$$1-{\gamma }^{\ast }+\frac{{\bar{a}}_V}{a_W}{\left(1-\frac{a_W}{{\bar{a}}_V+{\bar{a}}_W}\right)}^{\bar{d}}<\frac{a_W}{{\bar{a}}_V+{\bar{a}}_W}.$$

$${\mathcal{V}}_{\gamma ,\bar{d}}^{\text{run}}(x):=\sum _{k=0}^{\infty }{\gamma }^k{\ell }_{{\mathcal{U}}_{\gamma ,d(\varphi (k,x))}^{\ast }(\varphi (k,x))}(\varphi (k,x)),$$

$$V_{\gamma ,\infty }(x)\le V_{\gamma ,\bar{d}}^{\text{run}}(x)\le V_{\gamma ,\infty }(x)+w_{\gamma ,\bar{d}}\cdot \sigma (x),$$

$$V_{\gamma ,\bar{d}}^{\text{run}}(x):=\sum _{k=0}^{\infty }{\gamma }^k\ell (\varphi (k,x),u_k^r),$$

$$\begin{array}{rl}\ell (\varphi (k,x),u_k^r)& =V_{\gamma ,d(\varphi (k,x))}(\varphi (k,x))\\ & -\gamma V_{\gamma ,d(\varphi (k,x))-1}(\varphi (k+1,x)).\end{array}$$

$$\begin{array}{rl}{V}_{\gamma ,\bar{d}}^{\text{run}}(x)& \le V_{\gamma ,d(\varphi (0,x))}(\varphi (0,x))-\gamma V_{\gamma ,\bar{d}}(\varphi (1,x))\\ & +\gamma (V_{\gamma ,\infty }(\varphi (1,x))-\gamma V_{\gamma ,\bar{d}}(\varphi (2,x)))\\ & +{\gamma }^2(V_{\gamma ,\infty }(\varphi (2,x))-\gamma V_{\gamma ,\bar{d}}(\varphi (3,x)))+\dots \\ & =V_{\gamma ,d(\varphi (0,x))}(\varphi (0,x))\\ & +\sum _{k=1}^{\infty }{\gamma }^k(V_{\gamma ,\infty }(\varphi (k,x))-V_{\gamma ,\bar{d}}(\varphi (k,x))).\end{array}$$

$$\begin{array}{rl}{V}_{\gamma ,\bar{d}}^{\text{run}}(x)& \le V_{\gamma ,d(x)}(x)\\ & +\sigma (x)\frac{K{\bar{a}}_V({\bar{a}}_V+{\bar{a}}_W)}{a_W}{\left(1-\frac{a_W}{{\bar{a}}_V+{\bar{a}}_W}\right)}^{\bar{d}}\sum _{k=1}^{\infty }{\gamma }^k{e}^{-\lambda k},\\ & \le V_{\gamma ,\infty }(x)\\ & +\sigma (x)\frac{K{\bar{a}}_V({\bar{a}}_V+{\bar{a}}_W)}{a_W}{\left(1-\frac{a_W}{{\bar{a}}_V+{\bar{a}}_W}\right)}^{\bar{d}}\frac{\gamma }{e^{\lambda }-\gamma }.\end{array}$$

$$\frac{V_{\gamma ,\bar{d}}^{\text{run}}(x)-V_{\gamma ,\infty }(x)}{V_{\gamma ,\infty }(x)+W(x)}\le \frac{w_{\gamma ,\bar{d}}}{a_W}.$$

$$V_{\gamma ,d(v)}(v)\le \frac{V_{\gamma ,d(x)}(x)-{\ell }_{u_0^{\ast }}(x)}{\gamma }+{\gamma }^{\bar{d}}{\overline{\alpha }}_V(\sigma (\varphi (\bar{d},v,\hat{\mathbf{u}}{|}_{\bar{d}}))).$$

Full text

I Introduction

Optimistic planning (OP) is an algorithm that computes near-optimal control inputs for generic nonlinear discrete-time systems and infinite-horizon discounted costs, provided the set of inputs is finite, see [10, 13]. Given the current state, OP intelligently develops the tree of possible future states which are enumerable, as the input set is finite. By prioritizing branches with smaller costs, which are optimistic candidates to the infinite-horizon cost, OP efficiently exploits the available computational power. It then returns an optimal sequence of inputs for a finite-horizon discounted cost, where the horizon depends on the given computational budget and on the state. Guarantees on the mismatch between the obtained cost and the original infinite-horizon cost are provided in [10] and are of the form $\frac{\gamma^{d(x)}}{1-\gamma}$, where $\gamma\in(0,1)$ is the discount factor and $d(x)$ is the state-dependent horizon, which is related to the computation budget $B$.

OP is a priori well-suited for nonlinear switched discrete-time systems for which the control input corresponds to the switching signal [2]. While the (near)-optimal control of switched linear discrete-time systems is addressed in, e.g., [21, 20, 17, 4, 1], the case of nonlinear switched systems is still unraveling and concentrates on continuous-time systems, see e.g. [19, 22]. Even so, algorithms are often presented for a particular class of systems, consider finite-horizon optimality and ignore stability. There is therefore a need for tools for the (near-)optimal control of nonlinear switched systems. We propose a solution based on OP in this paper.

It appears that we cannot apply OP “off-the-shelf” adequately for optimal control problems. Indeed, OP requires that: (i) the stage cost takes value in $[0,1]$, which is not natural in control, as this excludes quadratic stage costs, for instance; (ii) the cost is discounted; (iii) the goal is to maximize the value function, and adapting OP to minimization is not straightforward. We therefore modify OP to overcome these limitations. We call this new algorithm OPmin. Similar to OP, OPmin returns a sequence of inputs, which minimizes a finite-horizon cost more efficiently than a brute-force approach (in general).

We make stabilizability and detectability assumptions, based on which we analyze the near-optimality guarantees of OPmin, that is, how the computed finite-horizon cost function compares to the infinite-horizon cost. The obtained bound on the mismatch between the two costs have the next desirable features: (i) it does not explode for $\gamma=1$, contrary to the bound in [10]; (ii) it decreases as the state is close to a given attractor, while the bound $\frac{\gamma^{d(x)}}{1-\gamma}$ in [10] is a constant for a constant horizon. In addition, inspired by our recent work [7, 6], we address the question of stability, which is ignored in [2, 10]. For this purpose, we rely on the same stabilizability and detectability assumptions as for the near-optimality analysis. We prove that a system, for which the inputs are generated by OPmin in a receding-horizon fashion, satisfies a semiglobal practical stability property, where the adjustable parameters are the computational budget of OPmin and the possible discount factor. We use a generic measuring function to define stability as in [8, 14, 6], thus covering point and set stability in a unified way. By strengthening the assumptions, we also derive a global exponential stability property. These stability results differ from our recent works in [7, 6] as the horizon here is state-dependent, and not fixed, like in [7, 6].

Finally, we investigate the relationship between the original infinite-horizon optimal value function, and the actual cost function obtained by applying OPmin in a receding-horizon fashion to the system, also known as running cost [9]. Assuming that the closed-loop system satisfies a global exponential stability property (for which we provide sufficient conditions, as mentioned above), we show that, indeed, increasing the horizon of optimization, i.e. the computational budget, implies that the running cost approaches the infinite-horizon optimal value function. Moreover, the mismatch between the running cost and the infinite-horizon cost decreases exponentially under mild conditions. We also provide the relative performance of the running cost to the infinite-horizon value function, similarly explored in [9]. Contrary to [9], we do not rely on relaxed dynamic programming assumptions for this purpose but on the aforementioned general stabilizability and detectability conditions. An example is provided to illustrate the theoretical results.

We think that this paper conveys an important message. It illustrates how an optimal algorithm from a different research field, namely artificial intelligence, can be adapted and tailored to solve an important control problem, here the near-optimal control of nonlinear switched discrete-time systems. It also demonstrates how control requirements, like stabilizability, detectability and stability, can be exploited to improve the original near-optimality guarantees of the algorithm.

It must be noted that tree-based algorithms have been considered in the literature for switched systems, albeit with different purposes. In [5], the stability of linear switched systems under arbitrarily switching is investigated for instance. The work in [12] considers a branch-and-bound approach for the discrete-time optimal control of switched linear systems and quadratic costs. On the other hand, (relaxed) dynamic approaches were considered in [15, 18]. In particular, [15] approximates the infinite-horizon optimal control problem for linear switched systems, and [18] develops a value iteration approach exploiting homogeneity of the system and stage costs. The main difference between our present paper and these references is that we address nonlinear switched systems and generic (discounted) costs.

The rest of the paper is organized as follows. Section II formally states the problem. OPmin is presented in Section III, and its near-optimality and stability properties are analyzed in Section IV. Section V provides an example.

Notation. Let $\mathbb{R}:=(-\infty,\infty)$, $\mathbb{R}_{\geq 0}:=[0,\infty)$, $\mathbb{Z}_{\geq 0}:=\{0,1,2,\ldots\}$ and $\mathbb{Z}_{>0}:=\{1,2,\ldots\}$. We use $(x,y)$ to denote $[x^{T},y^{T}]^{T}$, where $(x,y)\in\mathbb{R}^{n}\times\mathbb{R}^{m}$ and $n,m\in\mathbb{Z}_{>0}$. A function $\chi:\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ is of class $\mathcal{K}$ if it is continuous, zero at zero and strictly increasing, and it is of class $\mathcal{K}_{\infty}$ if it is of class $\mathcal{K}$ and unbounded. A continuous function $\beta:\mathbb{R}_{\geq 0}\times\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ is of class $\mathcal{KL}$ when $\beta(\cdot,t)$ is of class $\mathcal{K}$ for any $t\geq 0$ and $\beta(s,\cdot)$ is decreasing to 0 for any $s\geq 0$. The notation $\mathbb{I}$ stands for the identity map from $\mathbb{R}_{\geq 0}$ to $\mathbb{R}_{\geq 0}$. For any sequence $\bm{u}=[u_{0},u_{1},\dots]$ of length $d\in\mathbb{Z}_{\geq 0}\cup\{\infty\}$ where $u_{i}\in\mathbb{R}^{m}$, $i\in\{0,\ldots,d\}$, and any $k\in\{0,\ldots,d\}$, we use $\bm{u}|_{k}$ to denote the first $k$ elements of $\bm{u}$, i.e. $\bm{u}|_{k}=[u_{0},\dots,u_{k-1}]$ and $\bm{u}|_{0}=\varnothing$ by convention. Let $f\ :\ \mathbb{R}\to\mathbb{R}$, we use $f^{(k)}$ for the composition of function $f$ to itself $k$ times, where $k\in\mathbb{Z}_{\geq 0}$, and $f^{(0)}=\mathbb{I}$. The Euclidean norm of a vector $x\in\mathbb{R}^{n}$ is denoted by $|x|$. The distance of a vector $x\in\mathbb{R}^{n}$ to set $\cal A$ is defined as $|x|_{\cal A}=\inf\{|z-x|:z\in\cal A\}$.

II Problem Statement

Consider the system

[TABLE]

with state $x\in\mathbb{R}^{n}$, input $u\in\mathcal{U}$, where $\mathcal{U}:=\{1,\ldots,M\}$ is a finite set of admissible inputs with $M\geq 2$, and $f_{u}:\mathbb{R}^{n}\to\mathbb{R}^{n}$ for every input $u\in\mathcal{U}$. We use $\phi(k,x,\bm{u}|_{k})$ to denote the solution to system (1) at time $k\in\mathbb{Z}_{\geq 0}$ with initial condition $x$ and inputs $\bm{u}|_{k}=[u_{0},u_{1},\ldots,u_{k-1}]$, with the convention $\phi(0,x,\cdot)=\phi(0,x,\varnothing)=x$.

Our objective is to minimize the infinite-horizon cost

[TABLE]

where $x\in\mathbb{R}^{n}$ is the initial state, $\bm{u}$ is an infinite sequence of admissible inputs, $\ell_{u}:\mathbb{R}^{n}\to\mathbb{R}_{\geq 0}$ is the stage cost related to input $u\in\mathcal{U}$, and $\gamma\in(0,1]$ is the discount factor, which may be equal to $1$. Finding an infinite sequence of inputs which minimizes (2) is very difficult in general, as the case of linear switched systems with quadratic stage cost already shows [21]. We therefore aim at generating sequences of inputs that approximately minimize (2) instead, in a sense made precise in the following. For this purpose, we adapt optimistic planning (OP) as originally developed in [10] to be applicable for: (i) stage costs which are not constrained to take values in $[0,1]$, to cope with quadratic stage costs for instance; (ii) the undiscounted case, i.e. when $\gamma=1$ in (2); (iii) the minimization of (2), as opposed to maximization. We call this new algorithm OPmin. Furthermore, we also aim at ensuring stability properties for the induced closed-loop system. The algorithm is presented in the next section.

III OPmin

III-A Main idea

The algorithm we are going to present minimizes exactly, as we will prove, the following finite-horizon cost

[TABLE]

where $x\in\mathbb{R}^{n}$ is a given state, $d(x)\in\mathbb{Z}_{>0}$ is the horizon, which depends on $x$, and $\bm{u}=[u_{0},u_{1},...,u_{d(x)}]$ is a $d(x)+1$ sequence of admissible inputs. The associated optimal value function is

[TABLE]

and we define $\bm{u}_{\gamma,d(x)}^{*}$ an associated optimal input sequence, i.e. $V_{\gamma,d(x)}(x)=J_{\gamma,d(x)}(x,\bm{u}_{\gamma,d(x)}^{*}(x))$.

Compared to minimizing (2), problem (4) with finite $d(x)$ is solvable, as the input set $\mathcal{U}$ is finite. A brute-force approach can do it by developing all possible sequences. However, this is computationally intensive, in particular when $d(x)$ is large, as the computational cost grows exponentially with the horizon. OPmin, on the other hand, can intelligently explore the possible sequences to solve (4) with potentially larger $d(x)$ with the same computation, compared to a brute-force approach [10]. As we will show next, longer horizons imply smaller near-optimality bounds, and are therefore desirable.

III-B Algorithm

The objective of the algorithm is to find an input sequence such that (3) is minimized and $V_{\gamma,d(x)}(x)$ ‘approximates well’ $V_{\gamma,\infty}(x)$, as formalized later in Section IV. It does so by exploring the possible choices of inputs optimistically until the exhaustion of given computational resources. The computational resources available are denoted as a budget $B$, which corresponds to $B+1$ ‘leaf expansions’ (the root is expanded even at $B=0$). We denote by $\mathcal{T}$ the exploration tree from initial state $x\in\mathbb{R}^{n}$, constructed from admissible input sequences and their respective cost. A leaf is a node of $\mathcal{T}$ with no children, and the the set of all leaves of $\mathcal{T}$ is denoted $\mathcal{L}(\mathcal{T})$. At iteration $i\in\mathbb{Z}_{\geq 0}$, a leaf $L_{i}\in\mathcal{L}(\mathcal{T})$ is fully expanded. That is, for every $u\in\mathcal{U}$, we add a child to $L_{i}$ labeled by the resulting state $f_{u}(L_{i})$. We denote with a slight abuse of notation $\bm{u}(L_{i})$ the input sequence from the root to the state of leaf $L_{i}$. We denote by $J(L_{i}):=J_{\gamma,d(i)}(x,\bm{u}(L_{i}))$ cost (3) of the sequence that takes $x$ to the state of leaf $L_{i}$, with $d(i)=\text{depth}(L_{i})-1$, where depth is the number of edges (or inputs) from the root to $L_{i}$. The optimistic choice of leaf $L_{i}$ to expand is the leaf with minimal associated cost $J$ of all non-expanded leafs of $\mathcal{T}$. The algorithm is formalized next.

The most notable steps of Algorithm 1 are lines 4-5, where the optimistic exploration is realized. This optimistic choice guarantees that any sequence from descendants of a node $N$ will have costs $J$ greater than $N$, as $\ell\geq 0$. This implies that the first leaf to be expanded at a depth $d^{\prime}+1$ will be a suitable candidate for $V_{\gamma,d^{\prime}}(x)$, and $V_{\gamma,d(x)}(x)$ corresponds to the last suitable candidate calculated under budget $B$. Moreover, the expansion of the tree is independent from the ‘leaf selection’ step, and is fully determined by the optimistic selection of leaves. We have the following property for the returned leaf.

Proposition 1

Given a budget $B\geq 1$, Algorithm 1 terminates with output $S=\{\bm{u}_{\gamma,d(x)}^{*}(x),V_{\gamma,d(x)}(x)\}$ with horizon $d(x)\geq 0$. $\hfill\Box$

Proof:

Let $x\in\mathbb{R}^{n}$ and $B\geq 1$. We show that $S$ exactly calculates cost $V_{\gamma,d^{\prime}}(x)$ for some $d^{\prime}\in\mathbb{Z}_{>0}$. The optimal property of output $S$ to Algorithm 1 is fully determined in the particular iteration in which it is updated. Hence, let $\mathcal{T}_{i}$ be the tree to be expanded at iteration $i\in\mathbb{Z}_{\geq 0}$, in which $S$ is updated. We show now that the selected leaf $S$ with cost $J(S)$, where $J(S)$ is the cost associated to leaf $S$, attains the optimum of horizon $d^{\prime}:=\text{depth}(S)-1$, that is $J(S)=V_{\gamma,d^{\prime}}(x)$. Since $V_{\gamma,d^{\prime}}(x)\leq J(S)$ by the optimality of $V_{\gamma,d^{\prime}}(x)$, it suffices to prove $V_{\gamma,d^{\prime}}(x)\geq J(S)$. For this purpose, we proceed by contradiction, and we assume that $V_{\gamma,d^{\prime}}(x)<J(S)$. It follows from the fact that the input set $\mathcal{U}$ is finite that a sequence that attains the optimum $V_{\gamma,d^{\prime}}(x)$ exists, i.e. there is a node $N\neq S$, descendent of root $x$ and possibly not in $\mathcal{T}_{i}$, with cost $J(N)=V_{\gamma,d^{\prime}}(x)$. Since $\ell_{u}(x)\geq 0$ for any $x\in\mathbb{R}^{n}$ and $u\in\{1,\ldots,M\}$, any ancestor (parents, parents of parents and so on) of $N$ will have cost a lower cost than $J(N)$. Hence, let $L_{i}^{\prime}$ be the ancestor of $N$ such that $L_{i}^{\prime}\in\mathcal{L}(\mathcal{T}_{i})$, thus $J(L_{i}^{\prime})\leq J(N)$. Then, we have $J(L_{i}^{\prime})\leq J(N)=V_{\gamma,d^{\prime}}(x)<J(S)$, that is $J(L_{i}^{\prime})<J(S)$. However, $S$ is the optimistically chosen leaf $S=L_{i}$, and $J(S)=J(L_{i})\leq J(L)$ for any leaf $L\in\mathcal{L}(\mathcal{T}_{i})$, hence for leaf $L_{i}^{\prime}$, it follows that $J(L_{i}^{\prime})<J(S)\leq J(L_{i}^{\prime})$: we have attained a contradiction. Therefore, $J(S)\leq V_{\gamma,d^{\prime}}(x)$ and since $V_{\gamma,d^{\prime}}(x)\leq J(S)$, we conclude $V_{\gamma,d^{\prime}}(x)=J(S)$. Thus, at every update of $S$, a new optimal sequence is found with increased horizon $d^{\prime}\leftarrow d^{\prime}+1$. Furthermore, note that at iteration $i=1$, the ‘leaf selection’ step is guaranteed to be entered and, given budget $B\geq 1$, the outputs of Algorithm 1, $d$ and $S$, are fully determined. ∎

The horizon $d(x)$ in (4) depends on the given budget $B$, and will play a fundamental role in the near-optimality analysis provided later in Section IV. The next proposition provides a (conservative) relationship between budget $B$ and a given lower bound on $d(x)$.

Proposition 2

Given $\bar{d}\geq 0$ and budget $B\geq\frac{M^{\bar{d}+1}-1}{M-1}$, Algorithm 1 returns $S=\{\bm{u}_{\gamma,d(x)}^{*}(x),V_{\gamma,d(x)}(x)\}$ with horizon $B-1\geq d(x)\geq\bar{d}$. $\hfill\Box$

Proof:

The shallowest possible tree that can be explored with budget $B=\frac{M^{\bar{d}+1}-1}{M-1}$, under any circumstances, is the uniform, complete tree with depth $\bar{d}+1$, and one node of depth $\bar{d}+1$ expanded with children at depth $\bar{d}+2$. In this case $S$ is the node expanded at depth $\bar{d}+1$, hence $d(x)=\bar{d}$. Any other (e.g. optimistic) way of exploring the tree will lead to $d(x)\geq\bar{d}$. On the other hand, the deepest possible tree is the unbalanced tree, where the tree depth increase at every iteration, i.e. only one node is expanded per depth. Hence, given budget $B$, the maximum possible depth is $B$ and $d(x)=B-1$. Any other way of exploring the tree will lead to $d(x)\leq B-1$. We conclude $B-1\geq d(x)\geq\bar{d}$. ∎

Proposition 2 provides a relationship between a minimum desired horizon $\bar{d}$ in (3) and the required budget to achieve it. This relationship is derived from the worst-case exploration, which happens when OPmin is forced to uniformly explore the possible choices of switches for a given horizon. Due to the optimistic exploration, for given $B=\frac{M^{\bar{d}+1}-1}{M-1}$, $d(x)$ is often much larger in practice than $\bar{d}$. In the original OP study [10], this fraction is quantified by means of the branching factor $\kappa$, which we will investigate in future work.

IV Main results

In this section, we analyze the near-optimality properties of OPmin. We also provide conditions under which system (1), whose inputs are generated in a receding-horizon fashion by OPmin, exhibits stability properties. Assumptions are required for this purpose, which are now stated.

IV-A Assumptions

We first assume that the optimization goal (4) is well-posed in the following sense.

**Standing Assumption (SA) **

For any $x\in\mathbb{R}^{n}$, $\gamma\in(0,1]$, there exists an infinite sequence of admissible inputs $\bm{u}^{*}_{\gamma,\infty}(x)$, called optimal input sequence, which minimizes (2), i.e. $V_{\gamma,\infty}(x)=J_{\gamma,\infty}(x,\bm{u}_{\gamma,\infty}^{\boldsymbol{*}}(x))$ is finite. $\square$

General conditions to ensure SA can be found in [11].

We make the next general stabilizability and detectability assumptions on system (1) and stage cost $\ell$ as in [8, 14, 6], which are essential: (i) for the construction of near-optimality bounds of the algorithm; (ii) to ensure stability of the induced closed-loop system as demonstrated in the sequel.

Assumption 1

There exist $\overline{\alpha}_{V},\alpha_{W}\in\mathcal{K}_{\infty}$, continuous functions $W,\sigma:\mathbb{R}^{n}\to\mathbb{R}_{\geq 0}$, $\overline{\alpha}_{W}:\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ continuous, non-decreasing and zero at zero, such that the following conditions hold.

1. (i)

For any $x\in\mathbb{R}^{n}$, $\gamma\in(0,1]$,

[TABLE] 2. (ii)

For any $x\in\mathbb{R}^{n}$, $u\in\mathcal{U}$,

[TABLE]

$\hfill\square$**

Function $\sigma$ in Assumption 1 serves as a measuring function of the state and will be used to define stability, as in [14, 8, 6]. For instance, by defining $\sigma=|\cdot|$, one would be studying the stability of the origin, and by taking $\sigma=|\cdot|_{\cal A}$, one would study stability of set ${\cal A}\subset\mathbb{R}^{n}$. Item (i) is related to the stabilizability of system (1) with respect to stage cost $\ell$. Indeed, it is shown in [14, Lemma 1] that, if stage cost $\ell$ is uniformly globally exponentially controllable to zero with respect to $\sigma$ for system (1), see [8, Definition 2], then Assumption 1 is satisfied. On the other hand, item (ii) of Assumption 1 is a detectability property of the stage cost $\ell$ with respect to $\sigma$. For example, when $\ell_{u}(x)=\sigma(x)$, one verifies item (ii) of Assumption 1 with $W\equiv 0$ and $\alpha_{W}=\mathbb{I}$. For a more general view on Assumption 1, see the aforementioned references. Note that we do not require $\ell$ to take values in $[0,1]$ contrary to [10].

IV-B *Relationship between $V_{\gamma,d(x)}(x)$ and

$V_{\gamma,\infty}(x)$*

Algorithm 1 is able to calculate $V_{\gamma,d(x)}(x)$ exactly for any given $x\in\mathbb{R}^{n}$, however it is not obvious how $V_{\gamma,d(x)}(x)$ relates to $V_{\gamma,\infty}(x)$. Since $\ell$ is not constrained to take values in a given compact set, and we accept the undiscounted case, the tools used in [10] to analyze near-optimality are no longer applicable. We overcome this issue by exploiting Assumption 1.

Theorem 1

Suppose Assumption 1 holds. For any $x\in\mathbb{R}^{n}$, $\gamma\in(0,1]$ and $d(x)\in\mathbb{Z}_{>0}$,

[TABLE]

where $v_{\gamma,d(x)}(x):=\gamma^{d(x)}\overline{\alpha}_{V}\circ\underline{\alpha}_{Y}^{-1}\circ\left(\frac{\mathbb{I}-{\alpha}_{Y}\circ\overline{\alpha}_{Y}^{-1}}{\gamma}\right)^{(d(x))}\circ\overline{\alpha}_{Y}(\sigma(x))$. Here, ${\alpha}_{Y}=\underline{\alpha}_{Y}:=\alpha_{W},\overline{\alpha}_{Y}:=\overline{\alpha}_{V}+\overline{\alpha}_{W}$ and $\alpha_{W},\overline{\alpha}_{W},\overline{\alpha}_{V}$ come from Assumption 1. $\hfill\square$

Theorem 1 follows from [6, Theorem 3], therefore the proof is omitted. The lower-bound in (8) trivially holds from the optimality of $V_{\gamma,d(x)}(x)$ as $d(x)<\infty$, see Proposition 2. The upper-bound, on the other hand, implies that the infinite-horizon cost is at most $v_{\gamma,d(x)}(x)$ away from the finite-horizon $V_{\gamma,d(x)}(x)$. The error term $v_{\gamma,d(x)}(x)$ has three desirable properties compared to the term given in [10], which we recall is $\frac{\gamma^{d(x)}}{1-\gamma}$. First, when $\sigma(x)$ is small, so is $v_{\gamma,d(x)}(x)$. Second, $v_{\gamma,d(x)}(x)$ is finite for $\gamma=1$, while in [10], $\frac{\gamma^{d(x)}}{1-\gamma}\to\infty$ in this case. Third, $v_{\gamma,d(x)}(x)\to 0$ when $d(x)\to\infty$ for $\gamma$ sufficiently close to 1, as seen in [6, Lemma 3], which is true for OP only when $\gamma<1$. Thus, in contrast to OP, by exploiting stabilizability and detectability properties, we have obtained an error bound that forfeits the assumption $\ell\in[0,1]$, accepts the undiscounted case $\gamma=1$, and is decreasing in $d(x)$, even when $\gamma=1$.

Remark 1

Lemma 3 in [6] states that, when $\gamma\in(1-\frac{{\alpha}_{Y}(\Delta)}{\overline{\alpha}_{Y}(\Delta)},1]$ and $\sigma(x)\leq\Delta$ for any given $\Delta\geq 0$, the error $v_{\gamma,d(x)}(x)\to 0$ when $d(x)\to\infty$. In fact, it is shown in the proof of [6, Lemma 3] that for $\gamma\in(1-\frac{{\alpha}_{Y}(\Delta)}{\overline{\alpha}_{Y}(\Delta)},1]$, $v_{\gamma,d(x)}\leq\gamma^{d(x)}\overline{\alpha}_{V}\circ\underline{\alpha}_{Y}^{-1}(\overline{\alpha}_{Y}(\Delta))$, which has the same decrease rate in $d(x)$ as the original OP error bound $\gamma^{d(x)}\frac{1}{1-\gamma}$. However for $\gamma=1$ it follows that $v_{\gamma,d(x)}$ decreases as a function of $(\mathbb{I}-{\alpha}_{Y}\circ\overline{\alpha}_{Y}^{-1})^{(d(x))}$, which is not the case for OP under the assumptions of [10]. We can then control the size of $v_{\gamma,d(x)}$ as small as desired by chosing a suitable $d(x)$, via a suitable budget, and $\gamma$. $\hfill\Box$

It is unclear if increasing or decreasing $\gamma$ for a fixed $d(x)$ will increase or decrease error bound $v_{\gamma,d(x)}(x)$. Indeed, this is due to competing terms $\gamma^{d(x)}$, which increases as $\gamma$ increases for fixed $d(x)$, and $\left(\frac{\mathbb{I}-{\alpha}_{Y}\circ\overline{\alpha}_{Y}^{-1}}{\gamma}\right)^{(d(x))}$, which decreases as $\gamma$ increases with fixed $d(x)$. When Assumption 1 is satisfied with class $\mathcal{K}_{\infty}$ functions of a particular form, we show that the error term is uniform in $\gamma$, thus clarifying this issue.

Corollary 1

Suppose that Assumption 1 is satisfied and there exist $\bar{a}_{W}\geq 0$, $a_{W},\bar{a}_{V}>0$ such that $\overline{\alpha}_{V}(s)\leq\bar{a}_{V}\cdot s$, $\overline{\alpha}_{W}(s)\leq\bar{a}_{W}\cdot s$, $\alpha_{W}(s)\geq a_{W}\cdot s$ for any $s\geq 0$. Let $x\in\mathbb{R}^{n}$, $d(x)\in\mathbb{Z}_{>0}$, any $\gamma\in(0,1]$,

[TABLE]

where $\hat{v}_{d(x)}(x):=\displaystyle\frac{\bar{a}_{V}(\bar{a}_{V}+\bar{a}_{W})}{a_{W}}\left(1-\frac{a_{W}}{\bar{a}_{V}+\bar{a}_{W}}\right)^{d(x)}\!\!\!\!\sigma(x)$.

$\hfill\square$**

The proof of Corollary 1 is a direct substitution of the linear terms in Theorem 1, and is therefore omitted. Compared to Theorem 1, Corollary 1 provides an error bound $\hat{v}_{d(x)}(x)$ uniform in $\gamma$, linear in $\sigma(x)$, which also decreases exponentially to 0 for any $\gamma\in(0,1]$ when $d(x)\to\infty$, as a function111Indeed, $\left(1-\frac{a_{W}}{\bar{a}_{V}+\bar{a}_{W}}\right)\in[0,1)$ holds as $a_{W}s\leq\alpha_{W}(s)\leq\overline{\alpha}_{V}(s)+\overline{\alpha}_{W}(s)\leq(\bar{a}_{V}+\bar{a}_{W})s$ for any $s\geq 0$, in view of Corollary 1 and Proposition 3. of $\left(1-\frac{a_{W}}{\bar{a}_{V}+\bar{a}_{W}}\right)^{d(x)}$.

IV-C Stability

We now consider the scenario where (1) is controlled in a receding horizon fashion by OPmin as defined by Algorithm 1. That is, at each time instant $k\in\mathbb{Z}_{\geq 0}$, the first element of the optimal sequence $\bm{u}^{*}_{\gamma,d(x_{k})}(x_{k})$, is calculated by OPmin, and then applied to system (1). This leads to closed-loop system

[TABLE]

where $f_{\mathcal{U}^{*}_{\gamma,d(x)}(x)}(x)$ is the set $\{f_{u}(x):u\in\mathcal{U}^{*}_{\gamma,d(x)}(x)\}$, and \mathcal{U}_{\gamma,d(x)}^{*}(x):=\big{\{}u_{0}:\exists u_{1},\ldots,u_{d(x)}\in\mathcal{U}\text{ such that }V_{\gamma,d(x)}(x)=J_{\gamma,d(x)}(x,[u_{0},\ldots,u_{d(x)}])\big{\}} is the set of the first input of $d(x)$-horizon optimal input sequences at $x$. We denote by $\phi(k,x)$, with some abuse of notation, a solution to (10) at time $k\in\mathbb{Z}_{\geq 0}$ with initial condition $x\in\mathbb{R}^{n}$.

The next theorem provides stability guarantees for system (10).

Theorem 2

Consider system (10) and suppose Assumption 1 holds. There exists $\beta\in\mathcal{KL}$ such that for any $\delta,\Delta>0$, there exist $\gamma^{*}\in(0,1)$ and $\bar{d}\in\mathbb{Z}_{>0}$ such that for any $\gamma\in(\gamma^{*},1]$, any budget $B\geq\frac{M^{\bar{d}+2}-1}{M-1}$, any $x\in\{z\in\mathbb{R}^{n}\,:\,\sigma(z)\leq\Delta\}$, any solution $\phi(\cdot,x)$ to system (10) satisfies, for all $k\in\mathbb{Z}_{\geq 0}$

[TABLE]

$\hfill\square$**

The proof of Theorem 2 is given in the appendix. Theorem 2 provides a semiglobal practical stability property for set $\{z:\sigma(z)=0\}$. This implies that solutions to (10), with initial state $x$ such that $\sigma(x)\leq\Delta$, where $\Delta$ is any given (arbitrarily large) strictly positive constant, will converge to the set $\{z:\sigma(z)\leq\delta\}$, where $\delta$ is any given (arbitrarily small) strictly positive constant, by taking $\gamma$ sufficiently close to $1$ and a budget sufficiently large. Note that we take budget $B\geq\frac{M^{\bar{d}+2}-1}{M-1}$ to guarantee $d(x)>\bar{d}$, by Proposition 2. While OP was used in various control problems [2, 3, 16], such stability properties were never been proved before.

By strengthening Assumption 1, we can prove a global exponential stability property. Since it follows the same arguments as in [6, Corollary 2] and the modifications given in the appendix, the proof is omitted.

Corollary 2

Suppose that the conditions of Corollary 1 holds. Let $\gamma^{*},\bar{d}$ be such that

[TABLE]

Then, there exist $K,\lambda>0$, such that for any $\gamma\in(\gamma^{*},1]$, any budget $B\geq\frac{M^{\bar{d}+2}-1}{M-1}$, for any $x\in\mathbb{R}^{n}$, the solution $\phi(\cdot,x)$ to system (10), satisfies $\sigma(\phi(k,x))\leq K\sigma(x)e^{-\lambda k}$ for all $k\in\mathbb{Z}_{\geq 0}$. $\hfill\square$

Corollary 2 ensures a uniform global exponential stability property of $\{x:\sigma(x)=0\}$ for (10). Inequality (12) is always feasible for $\gamma^{*}$ sufficiently close to 1 and $\bar{d}$ sufficiently large. Indeed, we either first fix $\gamma^{*}\in(\bar{\gamma},1]$ with $\bar{\gamma}=1-\tfrac{a_{W}}{\bar{a}_{V}+\bar{a}_{W}}$ and then select $\bar{d}$ and the associated budget $B$ such that (12) holds, or we first fix budget $B$ with associated $\bar{d}>\tilde{d}$ with $\tilde{d}=\lfloor\tfrac{\ln(\bar{a}_{V}(\bar{a}_{V}+\bar{a}_{W})/a_{W}^{2})}{-\ln(1-\frac{a_{W}}{\bar{a}_{V}+\bar{a}_{W}})}\rfloor$ and select $\gamma^{*}$ such that (12) holds. The resulting pair $(\gamma^{*},\bar{d})$ and budget $B$ are suitable candidates for (12) by construction. This is consistent with results for finite-horizon discounted costs [6], where both $\gamma$ has to be sufficiently close to 1 and $\bar{d}$ has to be sufficiently large, and results for finite-horizon undiscounted cost [8], where $d$ has to be taken large.

Remark 2

It is possible to relax Corollary 2 conditions and derive semiglobal asymptotic results, similarly to [6, Corollary 1]. $\hfill\Box$

IV-D *Near-optimality

guarantees*

In Theorem 1, we have provided near-optimality guarantees of finite-horizon cost $V_{\gamma,d(x)}$ with respect to the infinite-horizon cost $V_{\gamma,\infty}(x)$. This is an important feature of OPmin, but this does not directly provide us with information on the actual value of the cost function (2) along solutions to (10). Indeed, we do not implement the whole sequence $\bm{u}^{*}_{\gamma,d(x)}(x)$ given by OPmin at $x$ in (10), instead we proceed in a receding horizon fashion. The relevant cost function to analyze is thus the running cost [9] defined as

[TABLE]

where $d(\phi(k,x))>\bar{d}$ for all $k\in\mathbb{Z}_{\geq 0}$, and $\bar{d}$ is a lower bound on the desired horizon at each step, which we can enforce by taking a sufficiently large budget according to Proposition 2. It has to be noted that $\mathcal{V}^{\text{run}}_{\gamma,\bar{d}}(x)$ is a set, since solutions of (10) are not necessarily unique. Each element $V_{\gamma,\bar{d}}^{\text{run}}(x)\in\mathcal{V}_{\gamma,\bar{d}}^{\text{run}}(x)$ corresponds then to the cost of a solution of (10). Clearly, $V_{\gamma,\bar{d}}^{\text{run}}(x)$ is not necessarily finite, as the stage costs may not decrease to 0 in view of Theorem 2. Indeed, only practical convergence is ensured in Theorem 2 in general. As a result, the corresponding running cost may not be finite. We therefore restrict our attention to the case where Corollary 2 holds, in the next theorem.

Theorem 3

Consider system (10) and assume that Corollary 2 holds with tuple $(K,\lambda,\gamma^{*},\bar{d})$. For any $\gamma\in(\gamma^{*},1]$, budget $B\geq\frac{M^{\bar{d}+2}-1}{M-1}$, $x\in\mathbb{R}^{n}$, and $V_{\gamma,\bar{d}}^{\text{run}}(x)\in\mathcal{V}_{\gamma,\bar{d}}^{\text{run}}(x)$,

[TABLE]

where $w_{\gamma,\bar{d}}:=\displaystyle\left(1-\frac{a_{W}}{\bar{a}_{V}+\bar{a}_{W}}\right)^{\bar{d}}\frac{K\bar{a}_{V}(\bar{a}_{V}+\bar{a}_{W})\gamma}{a_{W}(e^{\lambda}-\gamma)}.$ $\hfill\square$

Proof:

Let $x\in\mathbb{R}^{n}$, $\gamma\in(\gamma^{*},1]$, budget $B\geq\frac{M^{\bar{d}+2}-1}{M-1}$, and $\phi(k+1,x)\in F^{*}_{\gamma,d(\phi(k,x))}(\phi(k,x))$ for any $k\in\mathbb{Z}_{\geq 0}$ where $\phi$ is a solution to (10) initialized at $x$. For the sake of convenience, we denote $\ell(x,u):=\ell_{u}(x)$ for any $x\in\mathbb{R}^{n}$ and $u\in\mathcal{U}$. Consider

[TABLE]

where $u^{r}_{k}\in\mathcal{U}^{*}_{\gamma,d(\phi(k,x))}(\phi(k,x))$ such that $\phi(k+1,x)=f_{u^{r}_{k}}(\phi(k,x))$. Note that indeed $V_{\gamma,\bar{d}}^{\text{run}}(x)\in\mathcal{V}_{\gamma,\bar{d}}^{\text{run}}(x)$. The inequality $V_{\gamma,\infty}(x)\leq V^{\text{run}}_{\gamma,\bar{d}}(x)$ follows from optimality of $V_{\gamma,\infty}(x)$. Since $u^{r}_{k}\in\mathcal{U}^{*}_{\gamma,d(\phi(k,x))}(\phi(k,x))$, we derive from Bellman optimality principle that

[TABLE]

Since, for any $z\in\mathbb{R}^{n}$, $d(z)>\bar{d}$ holds for budget $B$ according to Proposition 2, $d(\phi(k,x))-1\geq\bar{d}$ and $V_{\gamma,d(\phi(k,x))-1}(\phi(k+1,x))\geq V_{\gamma,\bar{d}}(\phi(k+1,x))$ for all $k\in\mathbb{Z}_{\geq 0}$. As a result, in view of (16), $\ell(\phi(0,x),u^{r}_{0})\leq V_{\gamma,d(\phi(0,x))}(\phi(0,x))-\gamma V_{\gamma,\bar{d}}(\phi(1,x))$. On the other hand, since $V_{\gamma,d(\phi(k,x))}(\phi(k,x))\leq V_{\gamma,\infty}(\phi(k,x))$, $\ell(\phi(k,x),u^{r}_{k})\leq V_{\gamma,\infty}(\phi(k,x))-\gamma V_{\gamma,\bar{d}}(\phi(k+1,x))$. Thus

[TABLE]

According to Corollary 1, $V_{\gamma,\infty}(\phi(k,x))-V_{\gamma,\bar{d}}(\phi(k,x))\leq\hat{v}_{\bar{d}}(\phi(k,x))$, with $\hat{v}_{\bar{d}}(z)=\frac{\bar{a}_{V}}{a_{W}}\left(1-\frac{a_{W}}{\bar{a}_{V}+\bar{a}_{W}}\right)^{\bar{d}}(\bar{a}_{V}+\bar{a}_{W})\sigma(z)$ for any $z\in\mathbb{R}^{n}$. Hence, by direct substitution in (17), $V_{\gamma,\bar{d}}^{\text{run}}(x)\leq V_{\gamma,d(\phi(0,x))}(\phi(0,x))\quad+\tfrac{\bar{a}_{V}(\bar{a}_{V}+\bar{a}_{W})}{a_{W}}\left(1-\tfrac{a_{W}}{\bar{a}_{V}+\bar{a}_{W}}\right)^{\bar{d}}\sum_{k=1}^{\infty}\gamma^{k}\sigma(\phi(k,x))$. Recalling $\phi(0,x)=x$ and that $\sigma(\phi(k,x))\leq K\sigma(x)e^{-\lambda k}$ holds from Corollary 2, we obtain

[TABLE]

Since (18) holds for an arbitrary solution of (10), $\phi(k+1,x)=f_{u^{r}_{k}}(\phi(k,x))$ for any $k\in\mathbb{Z}_{\geq 0}$, (18) holds for any $V_{\gamma,\bar{d}}^{\text{run}}(x)\in\mathcal{V}_{\gamma,\bar{d}}^{\text{run}}(x)$. ∎

Similarly to Theorem 1, the inequality $V_{\gamma,\infty}(x)\leq V^{\text{run}}_{\gamma,\bar{d}}(x)$ of Theorem 3 directly follows from the optimality of $V_{\gamma,\infty}(x)$. On the other hand, the inequality $V_{\gamma,\bar{d}}^{\text{run}}(x)\leq V_{\gamma,\infty}(x)+w_{\gamma,\bar{d}}(x)$ provides a relationship between the running cost $V_{\gamma,\bar{d}}^{\text{run}}(x)$ and the infinite-horizon cost at state $x$, $V_{\gamma,\infty}(x)$. The term $w_{\gamma,\bar{d}}$ can be explicitly calculated, see Corollary 1 and [6, proof of Corollary 3] for the expressions of $K$ and $\lambda$. The latter inequality in (14) confirms the intuition coming from Theorem 1 that a large computational budget $B$ leads to tight near-optimality guarantees. That is, when $\bar{d}\to\infty$, or equivalently $B\to\infty$ according to Proposition 2, $w_{\gamma,\bar{d}}\to 0$ and $V^{\text{run}}_{\gamma,d}(x)\to V_{\gamma,\infty}(x)$, provided that $\gamma$ and budget $B$ have been chosen as to stabilize system (10). In contrast with Theorem 1, stability of system (10) plays a role in Theorem 3. Indeed, the term $K\frac{\gamma}{e^{\lambda}-\gamma}$ in (14) shows that the larger the exponential decay $\lambda$ is, the smaller the error term $w_{\gamma,\bar{d}}$ will be. The running cost for the original OP was considered in [3], and it was found to perform at worst like the finite sequence, i.e. $V_{\gamma,\bar{d}}^{\text{run}}(x)\leq V_{\gamma,\infty}(x)+\frac{\gamma^{\bar{d}}}{1-\gamma}$. Compared to the bound derived for OP, the bound in Theorem 3 has similar benefits as Theorem 1, namely we are not limited to $\ell\in[0,1]$, it does not explode to $\infty$ when $\gamma$ tends to 1, and when $\sigma(x)$ is small follows $w_{\gamma,\bar{d}}\cdot\sigma(x)$ small. Moreover, the mismatch decays exponentially in $\bar{d}$, independent on $\gamma$.

Remark 3

Inequality (14) can be written as a relationship of the finite-horizon costs in view of Theorem 3. In particular, we have $V_{\gamma,d(x)}(x)\leq V_{\gamma,\infty}(x)\leq V_{\gamma,\bar{d}}^{\text{run}}(x)\leq V_{\gamma,d(x)}(x)+w_{\gamma,\bar{d}}\cdot\sigma(x)$, for any $x\in\mathbb{R}^{n}$. Hence, $V_{\gamma,d(x)}(x)$ can be used to upper and lower bound $V_{\gamma,\bar{d}}^{\text{run}}(x)$ from the first call of OPmin at initial state $x$. $\hfill\Box$

Other authors have considered the running cost of finite-horizon controllers applied in receding horizon fashion, like [9] in the context of model predictive control. In particular, [9] derives relative performance of the running cost to the infinite-horizon optimal cost. From Theorem 3, we derive a similar result.

Corollary 3

Suppose Theorem 3 holds for system (10) with tuple $(K,\lambda,\gamma^{*},\bar{d})$. Then, for any $x\in\mathbb{R}^{n}$, $\gamma\in(\gamma^{*},1]$, budget $B\geq\frac{M^{\bar{d}+2}-1}{M-1}$, and $V_{\gamma,\bar{d}}^{\text{run}}(x)\in\mathcal{V}_{\gamma,\bar{d}}^{\text{run}}(x)$ such that $V_{\gamma,\infty}(x)>0$,

[TABLE]

$\hfill\Box$**

Proof:

Let $x\in\mathbb{R}^{n}$, $\gamma\in(\gamma^{*},1]$, budget $B\geq\frac{M^{\bar{d}+2}-1}{M-1}$. The proof follows by substitution of $a_{W}\sigma(x)\leq V_{\gamma,\infty}(x)+W(x)$ from item (i) of Proposition 3 in the appendix and $V_{\gamma,\bar{d}}(x)\leq V_{\gamma,\infty}(x)$ in $V_{\gamma,\bar{d}}^{\text{run}}(x)\leq V_{\gamma,\infty}(x)+w_{\gamma,\bar{d}}\cdot\sigma(x)$ from Theorem 3. ∎

Corollary 3 provides the relative performance of $V_{\gamma,\bar{d}}^{\text{run}}(x)+W(x)$ and $V_{\gamma,\infty}(x)+W(x)$. When Assumption 1 holds with $W\equiv 0$, Corollary 3 provides a relative relationship between any running cost $V_{\gamma,\bar{d}}^{\text{run}}$ and the infinite-horizon cost $V_{\gamma,\infty}$, as done in [9]. Interestingly, the obtained bound for relative performance is uniform in $x$. It does however conserve the desired properties of Theorem 3, in particular the exponential decay in $\bar{d}$ in view of the expression of $w_{\gamma,\bar{d}}$ in Theorem 3.

Corollary 3 follows from the stabilizability and detectability properties in Assumption 1, when the closed-loop system satisfies global exponential stability, while [9] results derive from relaxed dynamic programming, which relies on parameters of a modified Bellman optimality equation. Although a direct comparison between results is thus difficult, we note that our results are exponential in minimal horizon $\bar{d}$ with rate $\left(1-\frac{a_{W}}{\bar{a}_{V}+\bar{a}_{W}}\right)^{\bar{d}}$, while in [9] the obtained bound is of order $\frac{1}{(\eta+1)^{N}-1}$ for a parameter222We use $\eta$ instead of $\gamma$, as used in [9], to not be confused with our discount factor $\gamma$. $\eta$ derived from the relaxed dynamic programming property and constant horizon $N+1$. Hence, we expect that our bound may provide similar relative performance with a shorter horizon. We note that, while in [9] stability of the system is not assumed as in Corollary 3, a lower-bound for horizon $N+1$ such that the results of [9] holds is expected, as in Corollary 3.

V Example

We consider the cubic integrator from [8, Example 1], i.e. $x_{1}^{+}=x_{1}+u$, $x_{2}^{+}=x_{2}+u^{3}$ where $(x_{1},x_{2}):=x\in\mathbb{R}^{2}$ and $u\in\mathbb{R}$. It was verified in [8] that an open-loop sequence of inputs drives the system to $x=0$ in a finite number of steps. This open-loop sequence can be expressed as three feedback gains $K_{1}(x)=-x_{1}$, $K_{2}(x)=x_{2}^{\frac{1}{3}}$ and $K_{3}(x)=\left(-\frac{1}{2}+\sqrt{\frac{7}{12}}\right)x_{2}^{\frac{1}{3}}$, which are successively applied. We propose here to switch between these gains to minimize cost (2), with $\ell_{u}(x)=|x_{1}|^{3}+|x_{2}|+|K_{u}(x)|^{3}$ for any $x\in\mathbb{R}^{2}$ and $u\in\{1,2,3\}$. Note that we cannot design a local LQR controller for this system, due the lack of stabilizability of the linearized model at the origin. We therefore consider the switched system $x_{1}^{+}=x_{1}+K_{u}(x)$, $x_{2}^{+}=x_{2}+(K_{u}(x))^{3}$, for $u\in\{1,2,3\}$. We apply OPmin to illustrate the near-optimality and stability guarantees of Section IV. To do this, note that SA applies for the same reasons as in [8]. By taking $\sigma(x)=|x_{1}|^{3}+|x_{2}|$ for any $x\in\mathbb{R}^{2}$, Assumption 1 holds $\alpha_{W}=\mathbb{I}$, $W=\alpha_{W}=0$ and $\overline{\alpha}_{V}=14\mathbb{I}$, as in [8]. We verify Corollary 2 conditions with $a_{W}=1$, $\bar{a}_{V}=14$ and $\bar{a}_{W}=0$, and conclude that for $\gamma=1$, any budget $B\geq\frac{3^{73}-1}{2}$ ensures global exponential stability. Consequently, Theorem 3 also holds. The lower-bound on $B$ is conservative, as the horizon $\bar{d}$ itself in Corollary 2 is subject to some conservatism, and that OPmin will ensure large horizons for smaller budgets in general. We have thus fixed the budget to $B=3000$ for initial condition $x=[-1,1.5]^{\top}$. Figure 1 shows the evolution of the state, and we see that both $x_{1}$ and $x_{2}$ converge to zero, as ensured by Corollary 2. We then consider several initial conditions and we study the impact of the budget on the actual running cost estimated by running simulations over 200 steps. We see in Table I that the estimated running cost becomes smaller when increasing the budget, which is consistent with Theorem 3. In other words, the larger the budget, the better the running cost performance.

VI Conclusion

We have modified the optimistic planning algorithm in [10] to be applicable for the near-optimal, stable control of nonlinear switched discrete-time systems. We relied for this purpose on general stabilizability and detectability assumptions, originally stated in the model predictive control literature [8]. We have then analyzed the algorithm near-optimality guarantees, which has major features over the bound in [10] as discussed in Section IV. We have also shown that a system controlled in a receding-horizon fashion by OPmin satisfies stability properties. We have finally analyzed the mismatch between the optimal value function and the obtained running cost, and the same benefit as for the near-optimality guarantees were observed.

Appendix Proof of Theorem 2

The proof of Theorem 2 follows the same steps as the proof of [6, Theorem 2]. The difference is that the horizon in cost (4) is not fixed as in [6], but depends on the state. Nevertheless, as noted in [6, Remark 3], the results can be modified to hold for varying horizons, provided that considered horizons are lower-bounded by a sufficiently large constant, $\bar{d}$ in our case. This is what we explicitly show in the following. We first state the next Lyapunov properties.

Proposition 3

Suppose Assumption 1 holds. For any $\gamma\in(0,1]$ and $\bar{d}\in\mathbb{Z}_{>0}$, there exists $Y_{\gamma,d(\cdot)}:\mathbb{R}^{n}\to\mathbb{R}_{\geq 0}$ such that the following holds.

1. (i)

For any $x\in\mathbb{R}^{n}$, $\underline{\alpha}_{Y}(\sigma(x))\leq Y_{\gamma,d(x)}(x)\leq\overline{\alpha}_{Y}(\sigma(x)),$ where $\underline{\alpha}_{Y},\overline{\alpha}_{Y}$ come from Theorem 1. 2. (ii)

For any $x\in\mathbb{R}^{n}$ with $d(x)\geq\bar{d}$, $v\in F^{*}_{\gamma,d(x)}(x)$, Y_{\gamma,d(v)}(v)-Y_{\gamma,d(x)}(x)\leq\frac{1}{\gamma}\Big{(}-{\alpha}_{Y}(\sigma(x))+\Upsilon(Y_{\gamma,d(x)}(x),\gamma,\bar{d})\Big{)} where $\Upsilon(s,\gamma,\bar{d}):=(1-\gamma)s+\gamma^{\bar{d}}\overline{\alpha}_{V}\circ\underline{\alpha}_{Y}^{-1}\circ\left(\frac{\mathbb{I}-{\alpha}_{Y}\circ\overline{\alpha}_{Y}^{-1}}{\gamma}\right)^{(\bar{d})}(s)$, where $\overline{\alpha}_{V}$ comes from Assumption 1 and $\alpha_{Y}$ is defined in Theorem 1, and for any $s\geq 0$, $\Upsilon(s,\gamma,\bar{d})\to 0$ when $\gamma\to 1$ and $\bar{d}\to\infty$. $\hfill\square$

Proof:

The proof works by following the steps in [6, proof of Theorem 1], only one step needs to be carefully modified. To show the mechanism, we concentrate on the case where $\gamma<1$; the same reasoning applies when $\gamma=1$. Let $\gamma\in(0,1)$, $\bar{d}\in\mathbb{Z}_{>0}$, $x\in\mathbb{R}^{n}$ and $v\in F^{*}_{\gamma,d(x)}(x)$, with $d(x)\geq\bar{d}$ and $d(v)\geq\bar{d}$. There exists $[u^{*}_{0},u^{*}_{1},\ldots,u^{*}_{d(x)}]=\bm{u}^{*}_{\gamma,d(x)}(x)$ such that $v=f_{u^{*}_{0}}(x)$ and $\bm{u}^{*}_{\gamma,d(x)}(x)$ is an optimal input sequence for system (1) with cost (3). Hence $V_{\gamma,d(x)}(x)=J_{\gamma,d(x)}(x,\bm{u}^{*}_{\gamma,d(x)}(x))$.

We define $Y_{\gamma,d(x)}(x):=V_{\gamma,d(x)}(x)+W(x)$ for any $x\in\mathbb{R}^{n}$, where $W$ comes from Assumption 1. Since item (i) of Theorem 1 in [6] is valid for all $d\in\mathbb{Z}_{>0}$, item (i) of Proposition 3 holds. Now, consider the sequence $\hat{\bm{u}}:=[u^{*}_{1},u^{*}_{2},\ldots,u^{*}_{\bar{d}-1},\bar{\bm{u}}]$ where $\bar{\bm{u}}:=\bm{u}^{*}_{\gamma,\infty}(\phi({\bar{d}},x,\bm{u}^{*}_{\gamma,d(x)}(x)|_{\bar{d}}))$, $\bm{u}^{*}_{\gamma,d(x)}(x)|_{\bar{d}}=[u^{*}_{0},\ldots,u^{*}_{\bar{d}-1}]$ and $\phi$ denotes the solution of system (1). The sequence $\hat{\bm{u}}$ consists of the first $\bar{d}$ elements of $\bm{u}^{*}_{\gamma,d(x)}(x)$ after $u^{*}_{0}$, followed by an optimal input sequence of infinite length at state $\phi(\bar{d},x,\bm{u}_{\gamma,d}^{\boldsymbol{*}}(x)|_{\bar{d}})$. Note that such sequence only exists and is well defined if $d(x)\geq\bar{d}$, which is the case here, and that the sequence $\bar{\bm{u}}$ exists and minimizes $J_{\gamma,\infty}(\phi(\bar{d},x,\bm{u}_{\gamma,d}^{\boldsymbol{*}}(x)|_{\bar{d}}),\bar{\bm{u}}_{\bar{d}})$, which is guaranteed to exist by virtue of SA. From the definition of cost $J_{\gamma,d(\cdot)}$ in (3) and $V_{\gamma,d(v)}(v)$ in (4), $V_{\gamma,d(v)}(v)\leq J_{\gamma,d}(v,\hat{\bm{u}})=J_{\gamma,\bar{d}-1}(v,\hat{\bm{u}}|_{\bar{d}})+\gamma^{\bar{d}}J_{\gamma,\infty}(\phi(\bar{d},v,\hat{\bm{u}}|_{\bar{d}}),\bar{\bm{u}}).$ Then, by the same manipulations as in the proof of Theorem 1 in [6], we obtain

[TABLE]

We now bound $\sigma(\phi(\bar{d},v,\hat{\bm{u}}|_{\bar{d}}))$ by following the steps of [6]. In particular, we have that $Y_{\gamma,d(x)-(k+1)}(\phi(k+1,x,\bm{u}^{*}_{\gamma,d(x)}(x)|_{k+1}))\leq\left(\frac{\mathbb{I}-\alpha_{W}\circ\overline{\alpha}_{Y}^{-1}}{\gamma}\right)(Y_{\gamma,d(x)-k}(\phi(k,x,\bm{u}^{*}_{\gamma,d(x)}(x)|_{k}))$ for all $k\in\{1,\ldots,\allowbreak d(x)-1\}$, which yields by iteration, $Y_{\gamma,d(x)-(k+1)}(\phi(k+1,x,\bm{u}^{*}_{\gamma,d(x)}(x)|_{k+1}))\leq\left(\frac{\mathbb{I}-\alpha_{W}\circ\overline{\alpha}_{Y}^{-1}}{\gamma}\right)^{(k)}(Y_{\gamma,d(x)}(\phi(0,x,\varnothing)))$. Since $\sigma(\phi(\bar{d},v,\hat{\bm{u}}|_{\bar{d}}))=\allowbreak\sigma(\phi(\bar{d}+1,x,\bm{u}^{*}_{\gamma,d(x)}(x)|_{\bar{d}+1}))\leq\allowbreak\alpha_{W}^{-1}(Y_{\gamma,d(x)-(\bar{d}+1)}(\phi(\bar{d}+1,x,\bm{u}^{*}_{\gamma,d(x)}(x)|_{\bar{d}+1})))$ per item (i) of Proposition 3, it follows by fixing $k=\bar{d}$, that $\sigma(\phi(\bar{d},v,\hat{\bm{u}}|_{\bar{d}}))\leq\alpha_{W}^{-1}\left(\left(\frac{\mathbb{I}-\alpha_{W}\circ\overline{\alpha}_{Y}^{-1}}{\gamma}\right)^{(\bar{d})}(Y_{\gamma,d(x)}(x))\right)$. The desired result follows by applying the obtained upper-bound to (20) and noting that $\frac{V_{\gamma,d(x)}(x)}{\gamma}=V_{\gamma,d(x)}(x)+\frac{(1-\gamma)V_{\gamma,d(x)}(x)}{\gamma}$. The case where $\gamma=1$ is similarly treated as [6]. ∎

We can now finalize the proof of Theorem 2. Since budget $B\geq\frac{M^{\bar{d}+2}-1}{M-1}$, it follows that $d(x)>\bar{d}$ by Proposition 2. By following the proof of Theorem 2 in [6], the desired result is derived.