Optimal Stabilization Control for Discrete-time Markov Jump Linear System with Control Input Delay

TL;DR

This paper develops necessary and sufficient conditions for stabilizing and controlling Markov jump linear systems with input delay, using coupled algebraic Riccati equations, advancing the theoretical understanding of such systems.

Contribution

It introduces explicit stabilization conditions and designs an optimal controller for MJLS with input delay, using a novel Lyapunov equation and Riccati framework.

Findings

Stabilization conditions are both necessary and sufficient.

Optimal controller is derived via coupled algebraic Riccati equations.

Stabilizability is characterized by positive definite solutions under observability.

Abstract

This paper will investigate the infinite horizon optimal control and stabilization problems for the Markov jump linear system (MJLS) subject to control input delay. Different from previous works, for the first time, the necessary and sufficient stabilization conditions are explored under explicit expressions, and the optimal controller for infinite horizon is designed with a coupled algebraic Riccati equation. By introducing a new type of Lyapunov equation, we show that under the exact observability assumption, the MJLS with control input delay is stabilizable in the mean square sense with the optimal controller if and only if a coupled algebraic Riccati equation has a unique positive definite solution. The presented results are parallel to the optimal control and stabilization for standard system with input delay.

Equations

$$x(k+1)$$

$$J\triangleq \text{mbox}E\{\sum _{k=0}^{\infty }x(k{)}^{\prime }Qx(k)+\sum _{k=d}^{\infty }u(k-d{)}^{\prime }Ru(k-d)\},$$

$$u(k)=-K_{\theta (k)}^{0}x(k)-\sum _{i=1}^{d-1}{K}_{\theta (k)}^{i}u(k+i-d)$$

$$x(k+1)=A_{\theta (k)}x(k),y(k)=C_{\theta (k)}x(k),$$

$$y(k)\equiv 0,a.s.,\forall 0\le k\le N\Rightarrow x_0=0.$$

$$J_N$$

$$W_{l_{k-d}}(k-d)$$

$$T_{l_{k-d}}^0(k-d)$$

$$T_{l_{k-d}}^1(k-d)$$

$$T_{l_{k-d}}^j(k-d)$$

$$T_{l_{N-i}}^j(N-i)=0,$$

$$j=0,1,\cdots ,d-1,i=0,1,\cdots ,d-1,l_{N-i}\in \Theta ,$$

$$P_{l_{k-1}}(k-1)$$

$$P_{l_{k-1}}^0(k-1)$$

$$P_{l_N}(N)$$

$$P_{l_{N-i}}^0(N-i)$$

$$({\alpha }_{l_{k-1},l_{k-1}}^{d-1}(k-1,k-1){)}^{\prime }$$

$$({\alpha }_{l_{k-1},l_{k-j}}^{d-j}(k-1,k-j){)}^{\prime }$$

$$({\delta }_{l_{k-1}}^1(k-1){)}^{\prime }$$

$$({\delta }_{l_{k-1}}^j(k-1){)}^{\prime }$$

$$\text{mbox}E\{A_{l_k}^{\prime }({\alpha }_{l_k,l_k}^{d-1}(k,k){)}^{\prime }|{\mathcal{G}}_{k-1}\}$$

$$\text{mbox}E\{A_{l_k}^{\prime }({\alpha }_{l_k,l_{k-j+1}}^{d-j}(k,k-j+1){)}^{\prime }|{\mathcal{G}}_{k-1}\}$$

$$\text{mbox}E\{B_{l_k}^{\prime }({\alpha }_{l_k,l_k}^{d-1}(k,k){)}^{\prime }|{\mathcal{G}}_{k-1}\}$$

$$\text{mbox}E\{B_{l_k}^{\prime }({\alpha }_{l_k,l_{k-j+1}}^{d-j}(k,k-j+1){)}^{\prime }|{\mathcal{G}}_{k-j}\}$$

$$u(k-d)$$

$$W_{l_{k-d}}(k-d)>0,k=N,N-1,\cdots ,d,l_{k-d}\in \Theta .$$

$${\lambda }_{k-1}$$

$$J_N^{\ast }$$

$$W_{l_{k-d}}(k-d,N+1)>0,k=N+1,\cdots ,d.$$

$$W_{l_{k-1-d}}(k-1-d,N)=W_{l_{k-d}}(k-d,N+1)>0.$$

$$P_{l_{k-1}}(k-1,N)\ge 0,k=N,\cdots ,1,l_{k-1}\in \Theta ,$$

$$P_{l_{k-1}}^0(k-1,N)\ge 0,k=N,\cdots ,1,l_{k-1}\in \Theta ,$$

$$(P_{l_{k-1}}(k-1,N)-P_{l_{k-1}}^0(k-1,N))-\sum _{s=1}^{d-1}({\alpha }_{l_{k-1},l_{k-s}}^{d-s}(k-1,k-s,N){)}^{\prime }$$

$$\times W_{l_{k-1-s}}(k-1-s,N{)}^{-1}{\alpha }_{l_{k-1},l_{k-s}}^{d-s}(k-1,k-s,N)\ge 0,$$

$$k=N,\cdots ,d,l_{k-1},l_{k-d}\in \Theta .$$

$$(P_{l_{d-1}}(d-1,N_0)-P_{l_{d-1}}^0(d-1,N_0))-\sum _{s=1}^{d-1}({\alpha }_{l_{d-1},l_{d-s}}^{d-s}(d-1,d-s,N_0){)}^{\prime }$$

$$\times W_{l_{d-1-s}}(d-1-s,N_0{)}^{-1}{\alpha }_{l_{d-1},l_{d-s}}^{d-s}(d-1,d-s,N_0)>0,$$

$$l_{d-s}\in \Theta ,s=1,\cdots ,d-1.$$

$$W_{l_d}$$

Full text

Keywords: Optimal control, stabilization, algebraic Riccati equation, Markov jump linear system, input delay.

1 Introduction

Discrete-time MJLSs represent an important class of stochastic systems because they can be used to model random abrupt changes in structure. Dynamic systems with delays [1] or more general networked control applications [2], where communication networks are used to interconnect remote sensors, actuators and processors, have been shown to be amendable to MJLS modeling. Motivated by a wide spectrum of applications, there has been active research in the stabilization control problems for state delayed MJLSs[3]-[6], while no optimality of the controller was considered simultaneously. Different from the previous stabilization results, we mainly study the stabilization properties of the infinite horizon optimal controller in this paper.

The optimal control and stabilization problems for infinite horizon MJLSs have attracted the attention of many researchers. For example, necessary and sufficient existence conditions for the infinite horizon optimal controller were developed in [7], and sufficient stabilization conditions were explored which containing infinite sums. In [8], the definitions of the controllability and observability of discrete-time MJLS were developed. The existence of optimal steady-state controllers was guaranteed by the absolute controllability and the stability of the controlled system was guaranteed by the absolute observability. In [9], a necessary and sufficient condition was presented for the existence of a positive-semidefinite solution of the coupled algebraic Riccati-like equation (CARE) occurring in the infinite horizon optimal control problems. In [10], the concepts of mean square stabilizability and detectability were proposed, and a necessary and sufficient condition for the existence of the stabilizable solution to the infinite horizon optimal control problem was proposed under these concepts. In [11], a new detectability concept (weak detectability) for discrete-time MJLS was presented, and the new concept supplied a sufficient condition for the mean square stability of the infinite-horizon optimal controlled system. It has been shown in [11] that mean square detectability developed in [10] ensured weak detectability. Latter, Costa and do Val [12] summarized the available results and gave a proposition on the mean square stabilizable of the system. Under the assumption that the system was weak detectable, the system was mean square stabilizable if and only if there existed a positive semi-definite solution to the CARE. More recently, the optimal stationary output feedback control for MJLS waere studied in [13], [14]. It can be seen that no time delays are considered in the aforementioned results.

The optimal control and stabilization of linear system with input delay has received renewed interest in recent years. Various approaches have been developed. The prediction method is one of the famous methods to deal with input delays, which can be traced to the Smith predictor [15]. To overcome the limitation of the original Smith predictor that is just suitable for the open-loop stable system, the finite spectrum assignment and reduction method are developed in [16] and [17], respectively. It can be found that traditional prediction method contains the computation of distributed terms, which is computationally challenging and may be source of instability. To overcome this problem, the truncated predictor feedback [18] and closed-loop predictor approaches [19], [20], have been developed. Obviously, prediction method is an efficient tool for dealing with the optimal control and stabilization problems for the linear systems with input delay. However, it can’t be applied to the stochastic systems subject to input delay directly since the separation principle is not satisfied in the stochastic system. In [21] and [22], the stabilization problems for the discrete-time systems with multiplicative noise and input delays are considered. It has been shown that the system under consideration can be stabilized if and only if the algebraic Riccati-type equations have a unique solution such that a specific matrix is positive definite. It is worth mentioning that few results about infinite horizon optimal control and stabilization problem for MJLS with input delay have been published, which motivates us to undertake the present study. The fundamental questions we will answered in this paper are that: 1) To explore the necessary and sufficient condition for the existence of the infinite horizon optimal controller for the MJLS with input delay; 2) To find the mean square stablization condition for the optimal controlled MJLS.

In this paper, we aim to provide a thorough solution to the problems of optimal control and stabilization for infinite horizon discrete MJLS subject to input delay. As the preliminaries, the results of finite horizon optimal control for MJLS with control input delay are presented, and the necessary and sufficient solvability condition of finite horizon case is given in an explicit expression. By doing the convergence analysis on the coupled Riccati equation for the finite horizon case, the infinite horizon optimal controller and the stabilization conditions (necessary and sufficient) are derived. In addition, the Lyapunov function for stabilization is expressed with the optimal cost function. The stabilization result is obtained under the assumption of exact observability, under which it is shown that the optimal controlled MJLS is mean square stabilizable if and only if the coupled ARE has a unique positive definite solution.

The remainder of this paper is organized as follows. Section 2 provides the problem formulation and the preliminary results of finite horizon optimal control for MJLS with input delay. In Section 3, main results of the infinite horizon optimal control and stabilization problems are presented. Numerical examples are given in Section 4 to illustrate main results of this paper. Some concluding remarks are given in Section 5. Finally, relevant proofs are detailed in Appendices.

Notations: Throughout this paper, ${R}^{n}$ denotes the $n$-dimensional Euclidean space, $R^{m\times n}$ denotes the norm bounded linear space of all $m\times n$ matrices. For $L\in R^{n\times n}$, $L^{\prime}$ stands for the transpose of $L$. As usual, $L\geq 0(L>0)$ will mean that the symmetric matrix $L\in R^{n\times n}$ is positive semi-definite (positive definite), respectively. $\mbox{E}(.)$ denotes the mathematical expectation operator, $\mbox{P}(.)$ means the occurrence probability of an event. $\{\Omega,{\cal{G}},{\cal{G}}_{k},P\}$ represents a stochastic basis, with ${\cal{G}}_{k}$ the $\sigma$-field generated by the random variables $\{x(s),\theta(s);s=0,1,\cdots,k\}$. We will compactly write the sum $\sum_{l_{k-d+1}=1}^{L}\lambda_{l_{k-d},l_{k-d+1}}~{}\cdots~{}\sum_{l_{k}=1}^{L}\lambda_{l_{k-1},l_{k}}$ as $\Lambda_{l_{k-d},l_{k}}$ and denote $\sum_{l_{d-1}=1}^{L}\lambda_{l_{d},l_{d-1}}~{}\cdots~{}\sum_{l_{0}=1}^{L}\lambda_{l_{1},l_{0}}$ as $\Lambda_{l_{d},l_{0}}$.

2 Problem Formulation and Preliminaries

2.1 Problem Formulation

We consider in this paper the infinite horizon optimal control and stabilization problems for the MJLS with input delay. On the stochastic basis $(\Omega,{\cal{G}},{\cal{G}}_{k},\mbox{P})$, consider the following time-invariant MJLS

[TABLE]

where $x(k)\in{\mbox{R}}^{n}$ is the state, $u(k)\in{\mbox{R}}^{m}$ is the control input with delay $d>0$. $\theta(k)$ denotes the system mode which is a discrete-time homogeneous Markov chain. It is assumed that $\theta(k)$ takes values in a finite state space $\Theta\triangleq\{1,2,\cdots,L\}$ with transition probability matrix $\Pi\triangleq(\lambda_{ij})$, where $\lambda_{ij}\triangleq P(\theta(k+1)=j|\theta(k)=i)\geq 0$ for $i,j\in\Theta$ and $\sum_{j=1}^{L}\lambda_{ij}=1$ for every $i\in\Theta$. The matrices $A_{\theta(k)}$ and $B_{\theta(k)}$ are selected at each time step $k$ from time invariant sets $\{A_{1},\cdots,A_{L}\}$ and $\{B_{1},\cdots,B_{L}\}$ according to the value of the mode. We assume that $\theta(k)$ is independent of $x_{0}$ and $u(i),i=-d,\cdots,-1$, and the initial values $x_{0},u(i),i=-d,\cdots,-1$ are known.

The associated cost function is defined as

[TABLE]

where $Q$ and $R$ are deterministic symmetric weighting matrices with appropriate dimensions.

Now, we present some definitions.

Definition 1

The discrete-time MJLS (1) is said to be mean square stabilizable if for any initial conditions, there exist a ${\cal{G}}_{k}$-measurable constant gain controller

[TABLE]

such that $\lim_{k\rightarrow\infty}\mbox{E}(x(k)^{\prime}x(k))=0$.

Definition 2

The following MJLS

[TABLE]

is called exactly observable, if for any $N\geq n$,

[TABLE]

For simplicity, we rewrite system (3) as $(\bar{A},\bar{C})$, where $\bar{A}=(A_{1},\cdots,A_{L}),\bar{C}=(C_{1},\cdots,C_{L})$ with $C_{i}=C,(i=1,\cdots,L)$, and $Q=C^{\prime}C$.

Throughout this paper, the following assumptions are required to be satisfied.

Assumption 1

$R>0$* and $Q\geq 0$.*

Assumption 2

$(\bar{A},\bar{C})$* is exactly observable.*

Then the infinite-horizon optimal control and stabilization for system (1) can be stated as:

Problem 1: Find a ${\cal{G}}_{k}$-measurable controller $u(k)=-K_{\theta(k)}^{0}x(k)-\sum_{i=1}^{d-1}K_{\theta(k)}^{i}u(k+i-d)$, which minimizes the cost (2) and stabilizes system (1).

2.2 Preliminaries

In this part, we recall the finite horizon optimal control problem for system (1), which serves as the preparation for the infinite horizon case.

The finite horizon cost function associated to system (1) with admissible control law $u=(u(0),\cdots,u(N-d))$ is given by

[TABLE]

where $N>d$ is an integer, $Q,R$ and $P_{\theta(N+1)}$ are deterministic symmetric matrices with compatible dimension and $P_{\theta(N+1)}\geq 0$.

The finite-horizon optimal control for system (1) can be stated as: Find a ${\cal{G}}_{k}$-measurable $u(k)$ such that (4) is minimized subject to (1).

For the convenience of discussions latter, we denote a realization (sample path) of $\{\theta(k-d),\cdots,\theta(k)\}$ by $\{l_{k-d},\cdots,l_{k}\}$, and define a backward coupled difference equation

[TABLE]

for $k=N,N-1,\cdots,0,l_{k-d}\in\Theta$ with terminal values

[TABLE]

where $F_{l_{k},l_{i}}=A_{l_{k}}\cdots A_{l_{i}}(i=0,\cdots,k),F_{l_{k},l_{k+1}}=I$, and $P_{l_{k-1}}(k-1)$ and $P_{l_{k-1}}^{0}(k-1)$ satisfy the following backward recursions

[TABLE]

for $k=N,N-1,\cdots,0,l_{k-1}\in\Theta$ with terminal values

[TABLE]

(5)-(10) is termed as the backward coupled difference Riccati equation(CDRE). In addition, we introduce the following notations

[TABLE]

for $k=N,N-1,\cdots,0,l_{k-1},l_{k-j}\in\Theta$. The results for the finite horizon optimal control are stated as below.

Lemma 1

Consider $\alpha_{l_{k-1},l_{k-j}}^{d-j}(k-1,k-j)$ and $T_{l_{k-j}}^{j}(k-j)$ as in (6)-(8), (11) and (12), the following expressions are satisfied

[TABLE]

Proof. The detailed proof can be found in [24].

Theorem 1

Under Assumption 1, the finite-horizon optimal controller is given by

[TABLE]

and the controller is unique if and only if

[TABLE]

The optimal costate is

[TABLE]

and the optimal cost is

[TABLE]

Proof. The detailed proof can be found in [24].

Remark 1

The finite horizon optimal control result is obtained by two basic formulas: one is an improved delayed forward and backward jumping parameter equation (D-FBJPE) which is used to deal with the input delay, and the other is a d-step backward formula which is used to overcome the correlation of the jumping parameters. The detailed description can be seen from [24].

Remark 2

In order to make the time horizon $N$ explicit in the finite-time optimal control problem, we rewrite $W_{l_{k-d}}(k-d),T_{l_{k-d}}^{i}(k-d)(i=0,1,\cdots,d-1),P_{l_{k-1}}(k-1),P_{l_{k-1}}^{0}(k-1),\delta_{l_{k-1}}^{j}(k-1)(j=1,\cdots,d-1),\alpha_{l_{k-1},l_{k-j}}^{d-j}(k-1,k-j)(j=1,\cdots,d-1)$ as $W_{l_{k-d}}(k-d,N),T_{l_{k-d}}^{i}(k-d,N)(i=0,1,\cdots,d-1),P_{l_{k-1}}(k-1,N),P_{l_{k-1}}^{0}(k-1,N),\delta_{l_{k-1}}^{j}(k-1,N)(j=1,\cdots,d-1),\alpha_{l_{k-1},l_{k-j}}^{d-j}(k-1,k-j,N)(j=1,\cdots,d-1)$ respectively, and set $P_{l_{N+1}}(N+1)=0(l_{N+1}=1,\cdots,N)$.

3 Main Results

In this section, the main results of this paper will be presented, the necessary and sufficient stabilization conditions for optimal control systems will be established.

Before proposing the solution to Problem 1, the following lemmas will be given first.

Lemma 2

If $R>0$, the optimal control for the finite horizon case $(N\geq d)$ has a unique solution.

Proof. See Appendix A.

Remark 3

If $R>0$, we will obtain from Lemma 2 that

[TABLE]

It follows from (5)-(8) that $W_{l_{k-d}}(k-d,N+1)$ can be computed for $k=d-1,d-2,\cdots,0$ as well. Moreover, one yields from $P_{N+1}=0$ that

[TABLE]

So we have $W_{l_{k-d}}(k-d,N)>0$ for any $k=d-1,\cdots,0$ and $N$.

Lemma 3

Take $N\geq d$. If $R>0$, we have

[TABLE]

Proof. See Appendix B.

Lemma 4

If Assumptions 1 and 2 are satisfied, then there exists an integer $N_{0}\geq d$, such that

[TABLE]

Proof. See Appendix C.

In what follows, we introduce the generalized CAREs as

[TABLE]

[TABLE]

Then the main results of this paper can be stated as below.

Theorem 2

If Assumptions 1 and 2 are satisfied and system (1) is mean square stabilizable, we have the following properties:

1) When $N\rightarrow\infty$, $P_{l_{k-1}}(k-1,N),P_{l_{k-1}}^{0}(k-1,N),W_{l_{k-d}}(k-d,N),T_{l_{k-d}}^{s}(k-d,N)(s=0,1,\cdots,d-1)$ converge to $P_{l_{1}},P_{l_{1}}^{0},W_{l_{d}},T_{l_{d}}^{s}(s=0,1,\cdots,d-1)$ respectively for any $k\geq 0$ and $l_{k-1}\in\Theta$. Furthermore, $P_{l_{1}},P_{l_{1}}^{0},W_{l_{d}},T_{l_{d}}^{s}(s=0,1,\cdots,d-1)$ obey the coupled algebraic Riccati equations (26)-(35).

2)

[TABLE]

Proof. See Appendix D.

Theorem 3

If Assumptions 1 and 2 are satisfied, then system (1) is mean-square stabilizable if and only if there exists a unique solution to (26)-(35) such that

[TABLE]

In this case, the optimal controller given by

[TABLE]

stabilizes (1) and minimizes the performance index (2). The optimal value of (2) is given by

[TABLE]

Proof. See Appendix E.

4 Numerical Examples

In this section, we show the efficiency of the stabilization result for the infinite-horizon case. The specifications of the system (1) and the weighting matrices in (2) are as follows

[TABLE]

The initial distribution of $\theta(k)$ is $(0.5,~{}0.5)$s and probability transition matrix of $\theta(k)$ is \left[\begin{array}[]{cc}0.9&0.1\\ 0.3&0.7\\ \end{array}\right]. Meanwhile, the constant delay $d=2$, and the initial values of $x_{0},u(-1),u(-2)$ remain unchanged. we run $50$ Monte Carlo simulations, and select the first trajectory to show the efficiency of the proposed algorithm. By applying Theorem 2, the calculation results are listed as follow.

[TABLE]

And the stabilization condition

[TABLE]

is satisfied. If $\theta(k)=1$, the optimal stabilization controller is

[TABLE]

If $\theta(k)=2$, the optimal stabilization controller is

[TABLE]

From (38), we can derive the optimal cost $J_{0}=287.4952$.

5 Conclusion

In this paper, necessary and sufficient stabilization conditions for MJLS with input delay have been investigated. It is shown that, under the exact observability assumption, we show that the closed MJLS is mean square stabilizable if and only if the CARE admits a unique positive definite solution.

Appendix A Proof of Lemma 2

Proof. By induction method, we will show that the optimal control associated with performance (2) and system (1) has a unique solution for $N\geq d$. For $N=d$,

[TABLE]

It follows from

[TABLE]

that

[TABLE]

In light of Theorem 1, we know that the finite horizon optimal control admits a unique solution for $N=d$.

Next, suppose the solution to finite horizon control with terminal time $M=m$ is unique for some $m\geq d$, i.e.,

[TABLE]

Without loss of generality, we take $x(0)=0,u(-2)=u(-3)=\cdots=u(-d)=0$ and $u(-1)$ is an arbitrary value. Then we get that $x(1)=\cdots=x(d-1)=0$ and $x(d)=B_{\theta(d-1)}u(-1)$ is arbitrary. In view of (21), we can obtain the optimal cost with terminal time $m$

[TABLE]

Since $u(-1)$ is arbitrary, we have

[TABLE]

Recalling that the variables defined in (5)-(8) are time-invariant for $N$ owning to the selection that $P_{N+1}=0$, i.e.,

[TABLE]

For $N=m$, it yields from (82) that

[TABLE]

For $k=d$, we have

[TABLE]

It concludes from Theorem 1 that there exists a unique solution to the finite horizon optimal control with $N=m+1$. Then the uniqueness and existence of the solution to the optimal control for the finite horizon case is shown for $N\geq d$. This completes the proof.

Appendix B Proof of Lemma 3

Proof. In view of Theorem 1, we have that $W_{l_{k-d}}(k-d)>0$ for any $N\geq d$ and $d\leq k\leq N$. Therefore, we have from (10) that $P_{l_{k-1}}^{0}(k-1,N)\geq 0$ for $k=N,\cdots,1,l_{k-1}\in\Theta$. Recalling from the invariance of $P_{l_{k-1}}(k-1,N)=P_{l_{d-1}}(d-1,N-k+d)$ and (84), one yields

[TABLE]

It implies that $P_{l_{k-1}}(k-1,N)-P_{l_{k-1}}^{0}(k-1,N)\geq 0$. Then $P_{l_{k-1}}(k-1,N)\geq P_{l_{k-1}}^{0}(k-1,N)\geq 0$ follows immediately.

Next, we will show that (24) is satisfied. Let the system (1) start at $d$ with any initial value $x_{d}$ and denote it as

[TABLE]

In view of (21), the optimal value of (86) can be written as

[TABLE]

It follows from the arbitrary of $x(d)$ that

[TABLE]

Based on (88) and the time invariance of $P_{l_{k-1}}(k-1,N),P_{l_{k-1}}^{0}(k-1,N),\alpha_{l_{k-1},l_{k-s}}^{d-s}(k-1,k-s,N),W_{l_{k-s-1}}(k-s-1,N)$, one yields (24). This completes the proof of Lemma 3.

Appendix C Proof of Lemma 4

Proof. In order to facilitate the description, we denote

[TABLE]

If Assumption 1 is satisfied, one yields from Lemma 3 that $\Omega_{l_{d-1},l_{1}}(d-1,1,N)\geq 0$ for all $N\geq d$. In the next, we just need to show that there exists $N_{0}\geq d$ such that $\Omega_{l_{d-1},l_{1}}(d-1,1,N_{0})>0$. Assume this is not valid. Then we get an non-empty set

[TABLE]

In light of (86) and (87), we can deduce that $x^{\prime}\Omega_{l_{d-1},l_{1}}(d-1,1,N)x\leq x^{\prime}\Omega_{l_{d-1},l_{1}}(d-1,1,N+1)x$. Since $x$ is arbitrary, we get that $\Omega_{l_{d-1},l_{1}}(d-1,1,N)\leq\Omega_{l_{d-1},l_{1}}(d-1,1,N+1)$. Then if $x^{\prime}\Omega_{l_{d-1},l_{1}}(d-1,1,N+1)x=0$, we can deduce that $x^{\prime}\Omega_{l_{d-1},l_{1}}(d-1,1,N)x=0$. It implies that $X_{N+1}\subset X_{N}$. Note that each $X_{N}$ is non-empty and with finite-dimension, we can obtain that

[TABLE]

It follows from (89) that there must exist an integer $N_{1}$, such that for $N\geq N_{1}$, $\dim(X_{N})=\dim(X_{N_{1}})$ and thus $X_{N}=X_{N_{1}}$. It means that $\bigcap_{N\geq d}X_{N}=X_{N_{1}}\neq\emptyset$. Therefore, there must exist a nonzero vector $x\in X_{N_{1}}$, such that $x^{\prime}\Omega_{l_{d-1},l_{1}}(d-1,1,N+1)x=0$ for any $N\geq d$.

Set $x(d)=x$ in (87), we get

[TABLE]

It follows from the hypothesis $R>0$ and $Q=C^{\prime}C\geq 0$ that

[TABLE]

Then system (1) becomes as

[TABLE]

From the observability of (91), we get that $x(d)=0$. This contradicts the fact $x\neq 0$. So there exists some $N_{0}\geq d$ such that $\Omega_{l_{d-1},l_{d-s}}(d-1,d-s,N_{0})>0$. This completes the proof of Lemma 4.

Appendix D Proof of Theorem 2

Proof. (1) In the first part of the proof, we will show the convergence of the difference Riccati equations (5)-(8).

Now, we start to prove that $W_{l_{k-d}}(k-d,N),T_{l_{k-d}}^{s}(k-d,N)(s=0,1,\cdots,d-1)$ are convergent. Define

[TABLE]

then the system (1) can be rewritten as

[TABLE]

where

[TABLE]

and the performance index (2) becomes as

[TABLE]

where

[TABLE]

It can be seen that Problem 1 is equivalent to the minimization of (104) subject to (92). The necessary condition for minimizing the cost index of (104), i.e., the maximum principle, is given as

[TABLE]

Applying (105)-(107) and following a similar derivation as that of Theorem 1 in [21], we obtain the optimal controller

[TABLE]

and establish the relationship between $\bar{x}(k)$ and the costate $\bar{\lambda}_{k}$

[TABLE]

where $\bar{\Upsilon}_{l_{k}}(k),\bar{M}_{l_{k}}(k)$ and $\bar{P}_{l_{k}}(k)$ satisfy the following difference Riccati equations

[TABLE]

According to the partitioned form of the augmented state $\bar{x}(k)$, the block forms of $\bar{\lambda}_{k-1}$ and $\bar{P}_{l_{k}}(k)$ can be written as

[TABLE]

and (109) becomes as

[TABLE]

From (105), one yields

[TABLE]

In addition, from the maximum principle developed in [24], a necessary condition for minimizing (2) for system (1) is as:

[TABLE]

Compared (121) with (122), we get

[TABLE]

Based on the above expression, one yields

[TABLE]

Follow a similar discussion as (73)-(74) in [21], we can show that if system (1) is mean-square stabilizable, then the augmented system (92) is stabilizable in the mean square sense as well. Also, the observable keeps. So it follows from Theorem 2 in [23] that $\bar{P}_{l_{k}}^{(1,0)}(k),$ $\bar{P}_{l_{k}}^{(1,1)}(k),$ $\cdots,\bar{P}_{l_{k}}^{(1,d)}(k)$ are convergent. It concludes from the ergodicity of $\theta(k)$ and (123)-(126) that $T_{l_{k-1}}^{0}(k-1,N),\cdots,T_{l_{k-1}}^{d-1}(k-1,N),W_{l_{k-1}}(k-1,N)-R$ converge as well, i.e. $T_{l_{k-d}}^{0}(k-d,N-d+1),\cdots,T_{l_{k-d}}^{d-1}(k-d,N-d+1),W_{l_{k-d}}(k-d,N-d+1)-R$ are convergent. Denote $T_{l_{d}}^{0}=\lim_{N\rightarrow\infty}T_{l_{k-d}}^{0}(k-d,N-d+1),\cdots,T_{l_{d}}^{d-1}=\lim_{N\rightarrow\infty}T_{l_{k-d}}^{d-1}(k-d,N-d+1),W_{l_{d}}=\lim_{N\rightarrow\infty}W_{l_{k-d}}(k-d,N-d+1)$. Obviously, $T_{l_{d}}^{0},\cdots,T_{l_{d}}^{d-1},W_{l_{d}}$ satisfy (26)-(29).

Next, we will show the convergence of $P_{l_{k-1}}(k-1,N)$ and $P_{l_{k-1}}^{0}(k-1,N)$. In view of (10) and based on the convergence of $T_{l_{k-1}}^{0}(k-1,N),W_{l_{k-1}}(k-1,N)$, we get that $P_{l_{k-1}}^{0}(k-1)$ is convergent. Denote $P_{l_{1}}^{0}=\lim_{N\rightarrow\infty}P_{l_{k-1}}^{0}(k-1,N)$, then $P_{l_{1}}^{0}$ satisfies the algebraic Riccati equation (31). Now, we show the convergence of $P_{l_{k-1}}(k-1,N)$. For this purpose, we compute the cost $J_{N}$ with the initial values $u(-1)=u(-2)=\cdots=u(-d)=0$, but $x_{0}$ is arbitrary. First, define a Lyapunov function

[TABLE]

Applying (1) and (5)-(12), we deduce that

[TABLE]

The expressions (15)-(18) play a key role in the derivation of (128). As shown in Remark 5, (5)-(12) is also satisfied for $k=d-1,\cdots,0$. So (128) holds for $k=d-1,\cdots,0$. Adding from $k=0$ to $k=N$ on both sides of (128), one yields

[TABLE]

It follows from (129) that

[TABLE]

From the arbitrariness of $x_{0}$, we get

[TABLE]

which implies that $P_{l_{-1}}(-1,N)$ increases with respect to $N$. Analogous to the derivation of (73)-(77) in [21], we can show that there exist constant $\lambda$ and $c$, such that

[TABLE]

Recalling that $0\leq x_{0}^{\prime}P_{l_{-1}}(-1,N)x_{0}=J_{N}^{*}\leq J=2\lambda cx_{0}^{\prime}x_{0}$,which indicates that $0\leq P_{l_{-1}}(-1,N)\leq 2\lambda cI$. The boundedness of $P_{l_{-1}}(-1,N)$ is shown. Recalling that $P_{l_{-1}}(-1,N)$ is monotonically increasing. Therefore, it is convergent. In light of the time invariance of $P_{l_{k-1}}(k-1,N)$, we obtain that

[TABLE]

This completes the first part of the proof of Theorem 2.

(2) In the second part of the proof, we will show that (36) is satisfied. Note from Lemma 4, we known that there exists an integer $N_{0}$, such that

[TABLE]

On the other hand, $\Omega_{l_{d-1},l_{d-(d-1)}}(d-1,d-(d-1),N)$ is monotonically increasing with respect to $N$, so we have

[TABLE]

Therefore (36) is satisfied. The proof of Theorem 2 is completed.

Appendix E Proof of Theorem 3

Proof. Suppose $P_{l_{1}},P_{l_{1}}^{0},W_{l_{d}}$ and $T_{l_{d}}^{j}(j=0,1,\cdots,d-1)$ are the solutions to (26)-(35) such that

[TABLE]

In what follows, we shall prove that the optimal controller (37) stabilizes (1). For this purpose, we introduce the following Lyapunov function

[TABLE]

Employing (1) and (26)-(35), one gets

[TABLE]

(131) implies that $V(k,x(k))$ decreases with respect to $k$. Further, it follows from (130) that

[TABLE]

(132) indicates that $V(k,x(k))$ is bounded below. It follows from (131) and (132) that $V(k,x(k))$ is convergent.

Next, set $m$ to be any nonnegative integer. Via adding from $k=m+d$ to $k=m+N$ on both sides of (131) and letting $m\rightarrow+\infty$, we get that

[TABLE]

where the last equality holds owning to the convergence of $V(k,x(k))$. Note that

[TABLE]

By a time shift of length of $m$, it results in

[TABLE]

In view of (133), we get that

[TABLE]

Recalling from Lemma 4, we know that there exists an integer $N_{0}$, such that

[TABLE]

So (135) indicates that $\lim_{m\rightarrow\infty}\mbox{E}[x(m+d)^{\prime}x(m+d)]=0$. That is to say (37) stabilizes (1) in the mean-square sense.

In what follows, we will show that the cost function (2) is minimized by (37). Adding from $k=0$ to $k=N$ to (131), one yields

[TABLE]

Note that system (1) is mean square stabilizable, we have

[TABLE]

On the other hand,

[TABLE]

It concludes from (137) and (138) that

[TABLE]

Let $N\rightarrow\infty$ on both sides of (136), we get

[TABLE]

Since $W_{l_{d}}$ is positive, then (139) is minimized if and only if

[TABLE]

The corresponding optimal cost (38) is obtained.

From the derivation of Theorem 2, the following condition is satisfied

[TABLE]

The uniqueness can be similarly shown as the proof of Theorem 3 in [21], so we omit here. This completes the proof of Theorem 3.