Delayed Optimal Control of Stochastic LQ Problem

TL;DR

This paper investigates a stochastic discrete-time LQ control problem with delays and multiplicative noise, providing new Riccati-like equations and conditions for optimal control existence without definiteness constraints.

Contribution

It introduces the first Riccati-like equations and linear matrix inequalities for delayed stochastic LQ problems without definiteness constraints, characterizing optimal control conditions.

Findings

Existence of delayed optimal control is equivalent to solvability of Riccati-like equations.

New algebraic constraints and linear matrix inequalities are established for the problem.

Convexity properties of the cost functional are characterized via Riccati-like equations.

Abstract

A discrete-time stochastic LQ problem with multiplicative noises and state transmission delay is studied in this paper, which does not require any definiteness constraint on the cost weighting matrices. From some abstract representations of the system and cost functional, the solvability of this LQ problem is characterized by some conditions with operator form. Based on these, necessary and sufficient conditions are derived for the case with a fixed time-state initial pair and the general case with all the time-state initial pairs. For both cases, a set of coupled discrete-time Riccati-like equations can be derived to characterize the existence and the form of the delayed optimal control. In particular, for the general case with all the initial pairs, the existence of the delayed optimal control is equivalent to the solvability of the Riccati-like equations with some algebraic constraints, and both of them are also equivalent to the solvability of a set of coupled linear matrix equality-inequalities. Note that both the constrained Riccati-like equations and the linear matrix equality-inequalities are introduced for the first time in the literature for the proposed LQ problem. Furthermore, the convexity and the {uniform convexity} of the cost functional are fully characterized via certain properties of the solution of the Riccati-like equations.

Equations

$$\displaystyle\left\{\begin{array}[]{l}X_{k+1}=\big{(}A_{k}X_{k}+B_{k}u_{k}\big{)}+\big{(}C_{k}X_{k}+D_{k}u_{k}\big{)}w_{k},\\[2.84526pt] X_{t}=x,~{}~{}k\in\mathbb{T}_{t}\triangleq\{t,t+1,...,N-1\},~{}~{}t\in\mathbb{T}\triangleq\{0,1,...,N-1\},\end{array}\right.$$

$${\mathbb{E}}_{k+1}[w_{k+1}]=0,{\mathbb{E}}_{k+1}[(w_{k+1}{)}^2]=1,k\in \mathbb{T}.$$

$$\displaystyle J(t,x;u)=\sum_{k=t}^{N-1}\mathbb{E}\big{[}X_{k}^{T}Q_{k}X_{k}+u_{k}^{T}R_{k}u_{k}\big{]}+\mathbb{E}\big{[}X_{N}^{T}GX_{N}\big{]},$$

$$\displaystyle\mathcal{U}_{ad}^{t}=\big{(}l^{2}_{\mathcal{F}}(t;\mathbb{R}^{m})\big{)}^{d}\times l^{2}_{\mathcal{F}}(\mathbb{T}_{t}^{-d};\mathbb{R}^{m})$$

$$\displaystyle l^{2}_{\mathcal{F}}(t;\mathbb{R}^{m})=\Big{\{}\zeta\in\mathbb{R}^{m}\,\big{|}\,\zeta\mbox{ is }\mathcal{F}_{t}\mbox{-measurable, and }\mathbb{E}|\zeta|^{2}<\infty\Big{\}},~{}~{}t=0,...,N,$$

$$\displaystyle l^{2}_{\mathcal{F}}(\mathbb{T}^{-d}_{t};\mathbb{R}^{m})=\Big{\{}\nu=\{\nu_{k},k\in\mathbb{T}_{t}^{-d}\}\,\big{|}\,\nu_{k}\mbox{ is }\mathcal{F}_{k}\mbox{-measurable, and }\mathbb{E}|\nu_{k}|^{2}<\infty,k\in\mathbb{T}_{t}^{-d}\Big{\}}$$

$${\mathbb{T}}_t^{-d}=\{t,...,N-1-d\}.$$

$$J(t,x;\bar{u})=\underset{u\in {\mathcal{U}}_{ad}^t}{\inf }J(t,x;u).$$

$$\displaystyle\left\{\begin{array}[]{l}MM^{\dagger}M=M,~{}~{}M^{\dagger}MM^{\dagger}=M^{\dagger},\\ (MM^{\dagger})^{T}=MM^{\dagger},~{}~{}(M^{\dagger}M)^{T}=M^{\dagger}M.\end{array}\right.$$

$$\displaystyle\left[\begin{array}[]{cc}S&H^{T}\\ H&W\end{array}\right]\geq 0$$

$$S-H^T{W}^{†}H\ge 0,W\ge 0,W{W}^{†}H=H.$$

$$\displaystyle\left\{\begin{array}[]{l}{\Phi}(k,\ell)=(A_{k}+w_{k}C_{k})(A_{k-1}+w_{k-1}C_{k-1})\cdots(A_{\ell}+w_{\ell}C_{\ell}),~{}~{}k>\ell,\\[2.84526pt] \Phi(k,k)=A_{k}+w_{k}C_{k},\\[2.84526pt] {\Phi}(k,\ell)=I,k<\ell.\end{array}\right.$$

$$X_{k+1}=\Phi (k,t)x+\sum _{\ell =t}^k\Phi (k,\ell +1)(B_{\ell }+w_{\ell }{D}_{\ell })u_{\ell },k\in {\mathbb{T}}_t.$$

$$\displaystyle\left\{\begin{array}[]{l}\Gamma^{t}x=\Big{\{}\big{(}(\Gamma^{t}x)_{t},...,(\Gamma^{t}x)_{N-1}\big{)}\,\Big{|}\,(\Gamma^{t}x)_{k}={\Phi}(k-1,t)x,~{}k\in\mathbb{T}_{t}\Big{\}},\\[5.69054pt] \hat{\Gamma}^{t}x={\Phi}(N-1,t)x,\\ L^{t}u=\Big{\{}\big{(}(L^{t}u)_{t},...,(L^{t}u)_{N-1}\big{)}\,\Big{|}\,(L^{t}u)_{t}=0,(L^{t}u)_{k}=\displaystyle\sum_{\ell=t}^{k-1}\Phi(k-1,\ell+1)(B_{\ell}+w_{\ell}D_{\ell})u_{\ell},~{}k\in\mathbb{T}_{t+1}\Big{\}},\\ \hat{L}^{t}u=\displaystyle\sum_{\ell=t}^{N-1}\Phi(N-1,\ell+1)(B_{\ell}+w_{\ell}D_{\ell})u_{\ell}.\end{array}\right.$$

$$X_k=({\Gamma }^{t}x{)}_k+(L^{t}u{)}_k,k\in {\mathbb{T}}_t,$$

$$X_N={\hat{\Gamma }}^{t}x+{\hat{L}}^{t}u.$$

$$\displaystyle\left\{\begin{array}[]{l}\Gamma^{t}:l^{2}_{\mathcal{F}}(t;\mathbb{R}^{n})\mapsto l^{2}_{\mathcal{F}}({\mathbb{T}}_{t};\mathbb{R}^{n}),~{}~{}\hat{\Gamma}^{t}:l^{2}_{\mathcal{F}}(t;\mathbb{R}^{n})\mapsto l^{2}_{\mathcal{F}}(N;\mathbb{R}^{n}),\\ L^{t}:\mathcal{U}^{t}_{ad}\mapsto l^{2}_{\mathcal{F}}({\mathbb{T}}_{t};\mathbb{R}^{n}),~{}~{}\hat{{L}}^{t}:\mathcal{U}^{t}_{ad}\mapsto l^{2}_{\mathcal{F}}(N;\mathbb{R}^{n})\end{array}\right.$$

$$\displaystyle\left\{\begin{array}[]{l}V_{k}=A_{k}^{T}\mathbb{E}_{k}V_{k+1}+C_{k}^{T}\mathbb{E}_{k}(V_{k+1}w_{k})+\xi_{k},\\[2.84526pt] V_{N}=\eta,~{}~{}k\in\mathbb{T}_{t}.\end{array}\right.$$

$${\Gamma }^{t\ast }\xi =V_t^0,$$

$$(L^{t\ast }\xi {)}_k=B_k^T{\mathbb{E}}_{k-d}{V}_{k+1}^0+D_k^T{\mathbb{E}}_{k-d}(V_{k+1}^0{w}_k),k\in {\mathbb{T}}_t,$$

$${\hat{\Gamma }}^{t\ast }\eta =V_t^{00},$$

$$({\hat{L}}^{t\ast }\eta {)}_k=B_k^T{\mathbb{E}}_{k-d}{V}_{k+1}^{00}+D_k^T{\mathbb{E}}_{k-d}(V_{k+1}^{00}{w}_k),k\in {\mathbb{T}}_t.$$

$$\displaystyle\mathbb{E}\big{[}\eta^{T}X_{N}-V_{t}^{T}x\big{]}=\sum_{k=t}^{N-1}\mathbb{E}\Big{[}V_{k+1}^{T}X_{k+1}-V_{k}^{T}X_{k}\Big{]}$$

$$\displaystyle\hphantom{\mathbb{E}\big{[}\eta^{T}X_{N}-V_{t}^{T}x\big{]}}=\sum_{k=t}^{N-1}\mathbb{E}\Big{[}\big{(}A_{k}^{T}V_{k+1}+C_{k}^{T}(V_{k+1}w_{k})-V_{k}\big{)}^{T}X_{k}\Big{]}$$

$$\displaystyle\hphantom{\mathbb{E}\big{[}\eta^{T}X_{N}-V_{t}^{T}x\big{]}=}+\sum_{k=t}^{N-1}\mathbb{E}\Big{[}\big{(}B_{k}^{T}V_{k+1}+D_{k}^{T}(V_{k+1}w_{k})\big{)}^{T}u_{k}\Big{]}$$

$$\displaystyle\hphantom{\mathbb{E}\big{[}\eta^{T}X_{N}-V_{t}^{T}x\big{]}}=-\sum_{k=t}^{N-1}\mathbb{E}\big{[}\xi_{k}^{T}X_{k}\big{]}+\sum_{k=t}^{N-1}\mathbb{E}\Big{[}\big{(}B_{k}^{T}\mathbb{E}_{k-d}V_{k+1}+D_{k}^{T}\mathbb{E}_{k-d}(V_{k+1}w_{k})\big{)}^{T}u_{k}\Big{]}.$$

$$\displaystyle\langle\Gamma^{t}x,\xi\rangle=\sum_{k=t}^{N-1}\mathbb{E}\big{[}\xi_{k}^{T}(\Gamma^{t}x)_{k}\big{]}=\sum_{k=t}^{N-1}\mathbb{E}\big{[}\xi_{k}^{T}X_{k}\big{]}=\mathbb{E}\big{[}x^{T}V_{t}^{0}\big{]}=\langle x,V^{0}_{t}\rangle,$$

$$\displaystyle\langle L^{t}u,\xi\rangle=\sum_{k=t}^{N-1}\mathbb{E}\big{[}(L^{t}u)_{k}^{T}\xi_{k}\big{]}=\sum_{k=t}^{N-1}\mathbb{E}\big{[}X_{k}^{T}\xi_{k}\big{]}=\sum_{k=t}^{N-1}\mathbb{E}\Big{[}u_{k}^{T}\big{(}B_{k}^{T}\mathbb{E}_{k-d}V^{0}_{k+1}+D_{k}^{T}\mathbb{E}_{k-d}(V^{0}_{k+1}w_{k})\big{)}\Big{]}.$$

$$\displaystyle\langle\hat{\Gamma}^{t}x,\eta\rangle=\mathbb{E}\big{[}\eta^{T}X_{N}\big{]}=\mathbb{E}\big{[}x^{T}V^{00}_{t}\big{]}=\langle x,V^{00}_{t}\rangle.$$

$$\displaystyle\langle\hat{L}^{t}u,\eta\rangle=\mathbb{E}\big{[}\eta^{T}X_{N}\big{]}=\sum_{k=t}^{N-1}\mathbb{E}\Big{[}u_{k}^{T}\big{(}B_{k}^{T}\mathbb{E}_{k-d}V^{00}_{k+1}+D_{k}^{T}\mathbb{E}_{k-d}(V^{00}_{k+1}w_{k})\big{)}\Big{]}.$$

$$\displaystyle\left\{\begin{array}[]{l}(QX)_{k}=Q_{k}X_{k},~{}k\in\mathbb{T}_{t},~{}\forall X\in l^{2}_{\mathcal{F}}({T}_{t};\mathbb{R}^{n}),\\ (Ru)_{k}=R_{k}u_{k},~{}k\in\mathbb{T}_{t},~{}\forall u\in\mathcal{U}^{t}_{ad}.\end{array}\right.$$

$$J(t,x;u)=⟨Q({\Gamma }^{t}x+L^{t}u),{\Gamma }^{t}x+L^{t}u{⟩}_{l_{\mathcal{F}}^2(T_t;{\mathbb{R}}^n)}+⟨Ru,u{⟩}_{{\mathcal{U}}_{ad}^t}+⟨G({\hat{\Gamma }}^{t}x+{\hat{L}}^{t}u),{\hat{\Gamma }}^{t}x+{\hat{L}}^{t}u{⟩}_{l_{\mathcal{F}}^2(N;{\mathbb{R}}^n)}$$

$$=⟨{\Theta }_1^{t}u,u{⟩}_{{\mathcal{U}}_{ad}^t}+2⟨{\Theta }_2^{t}x,u{⟩}_{{\mathcal{U}}_{ad}^t}+⟨{\Theta }_3^{t}x,x{⟩}_{l_{\mathcal{F}}^2(t;{\mathbb{R}}^n)}$$

Full text

Delayed Optimal Control of Stochastic LQ Problem

††thanks: Part of this paper was presented at the 36th Chinese Control Conference.

Yuan-Hua Ni     Cedric Ka-Fai Yiu     Huanshui Zhang     Ji-Feng Zhang College of Computer and Control Engineering, Nankai University, Tianjin 300350 , P. R. ChinaDepartment of Applied Mathematics, The Hong Kong Polytechnic University, Hunghom, Kowloon, Hong Kong, P.R. China.School of Control Science and Engineering, Shandong University, Jinan 250061, P.R. China.Key Laboratory of Systems and Control, Institute of Systems Science, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, and the School of Mathematical Sciences, University of Chinese Academy of Sciences, P. R. China.

Abstract: A stochastic LQ problem with multiplicative noises and transmission delay is studied in this paper, which does not require any definiteness constraint on the cost weighting matrices. From some abstract representations of the system and cost functional, the solvability of this LQ problem is characterized by some conditions with operator form. Based on these, necessary and sufficient conditions are derived for the case with a fixed time-state initial pair and the general case with all the time-state initial pairs. For both cases, a set of coupled discrete-time Riccati-like equations can be derived to characterize the existence and the form of the delayed optimal control. In particular, for the general case with all the initial pairs, the existence of the delayed optimal control is equivalent to the solvability of the Riccati-like equations with some algebraic constraints, and both of them are also equivalent to the solvability of a set of coupled linear matrix equality-inequalities. Note that both the constrained Riccati-like equations and the linear matrix equality-inequalities are introduced for the first time in the literature for the proposed LQ problem. Furthermore, the convexity and the uniform convexity of the cost functional are fully characterized via certain properties of the solution of the Riccati-like equations.

Key words: stochastic linear-quadratic optimal control, transmission delay, forward-backward stochastic difference equation, convexity

AMS subject classifications. 49N10, 49N35, 93E20

1 Introduction

Linear-quadratic (LQ, for short) optimal control was pioneered by Kalman [18] in 1960, which is now a classical yet fundamental problem in control theory. Extension to stochastic LQ problems was first carried out by Wonham [38] in 1968, and has received considerable interests and efforts since then. A common assumption of most literature on stochastic LQ problems is that the state weighting matrices are nonnegative definite and the control weighting matrices are positive definite. Contrary to this, Chen, Li and Zhou [11] revealed in 1998 that a stochastic LQ problem with multiplicative noises might still be solvable even if the cost weighting matrices are indefinite. More about this kind of LQ problems can be found in [2] [5] [15] [30] and references therein. Recently, some researchers are interested in the so-called mean-field LQ problems [25] [26] [31] [36] [41] [42]. An important feature of mean-field control problems is that the expected values of the state and control enter nonlinearly into the cost functional, which will bring new phenomena and new theoretical difficulties.

Note that all the aforementioned papers are free of time delay. If time delay happens to appear in the system state, the control input or the information-transmission channel, it is much more complicated and challenging to design the optimal control of the corresponding LQ problems. Such kind of LQ problems have been extensively studied since 1970’s; see, for example, [6] [12] [19] [34] [44] or other related literature [7] [20] [23] [32]. Concerned with a deterministic LQ problem with input delay, it is shown [34] that the delayed optimal control is obtained by invoking the Smith predictor theory, and that the optimal gains are same to those of the LQ problem without input delay. Unfortunately, the results about deterministic LQ problems (with input delay) cannot be directly generalized to the stochastic setting. In [44], the authors considered a discrete-time stochastic LQ problem with input delay and multiplicative noises, and showed that the optimal control (if exists) is a linear feedback of $d$-step-lagged conditional expectation of current states and that the optimal gains are computed via a set of coupled discrete-time Riccati-like equations. Here, the set of discrete-time Riccati-like equations differs significantly from what we have in hand the standard discrete-time Riccati equation.

It is worth pointing out that the stochastic systems with multiplicative noises have been extensively studied in the past half century. From the viewpoint of mathematics, almost all the theories about stochastic differential equations (SDEs, for short) are for the case with multiplicative noises, and there are lots of practical motivations to study such kind of SDEs. The study of controlled systems with multiplicative noises is also popular in the control community; a recent small collection in the literature related to our paper includes [2] [5] [9] [10] [11] [15] [16] [25] [27] [30] [37] [41].

In this paper, a general discrete-time stochastic LQ problem with multiplicative noises and transmission delay is thoroughly investigated, whose cost weighting matrices for the state and control are allowed to be indefinite. Apart from intending to generalize the existing results [6] [12] [19] [34] [44] to the joint case with indefiniteness and time delay, the topic of this paper is also partially motivated by recent progresses in network control system and other related areas. Transmission delay, or sometimes called as communication delay, is a key feature of network control systems [8] [14] [35], which is generally caused by the limited bit rate of communication channels. In fact, transmission delay has been extensively studied in the areas such as discrete-event dynamic systems [46], multi-agent systems [22] [33] [21], networked mobile robots [39], receding horizon control [17], flexible spacecraft [13], and so on. Furthermore, such kind of delays are also related to the measurement delays [3] [4] [24] [29] [47], which arise in measurement channels.

The contents of this paper are as follows. For the completeness and parallel to that in [43], the considered problem (Problem (LQ)) is converted in Section 3 to a quadratic optimization problem in the Hilbert space. By this reformulation, we can derive some abstract conditions on the solvability of Problem (LQ), which gives us an overall perspective of Problem (LQ) and motivates the analysis of the sections followed. This part of work is a discrete-time version (with state transmission delay) of the results in [43], and the backward stochastic difference equations (BS$\Delta$Es, for short) are involved here.

In Section 4, for the case with a fixed time-state initial pair, the solvability of Problem (LQ) at that initial pair is equivalent to that a stationary condition and a convexity condition are satisfied, with the backward state of a forward-backward stochastic difference equation (FBS$\Delta$E, for short) being involved in the stationary condition. Further, a set of coupled discrete-time Riccati-like equations is introduced, by which we can express the backward state of the FBS$\Delta$E via its forward state. Moreover, equivalent characterizations of the stationary condition and the convexity condition are derived via certain properties of the solution of the Riccati-like equations.

In Section 5, for the case with all the time-state initial pairs, the following facts are shown to be equivalent: (i) Problem (LQ) is finite; (ii) Problem (LQ) is solvable; (iii) a set of constrained coupled discrete-time Riccati-like equations is solvable; (iv) a set of coupled linear matrix equality-inequalities (LMEIs, for short) is solvable. Moreover, the unique solvability of Problem (LQ) at the initial pair $(t,x)$ is shown to be equivalent to the unique solvability at any initial pair $(k,\xi)\in\{t,...,N-1\}\times\mathbb{R}^{n}$, both of which are equivalent to the uniform convexity of the cost functional and the positive definiteness of certain matrices involved in the constrained Riccati-like equations.

From our derived results, we have the following remarks.

• •

For Problem (LQ), the case with a fixed time-state initial pair differs significantly from the case with all the time-state initial pairs; this can be seen from Theorem 4.12 and Theorem 5.4. Hence, we separately discuss the two cases.

• •

By the stationary condition and a backward procedure of calculations, we can get the Riccati-like equations (4.12)-(4.22) and express FBS$\Delta$E’s backward state via its forward state and the solution of the Riccati-like equations. Due to the $d$-step-lagged information structure, the Riccati-like equations are much more complicated than the standard discrete-time Riccati equation.

• •

The convexity of the cost functional is fully characterized in Theorem 4.9 via certain properties of solution of the Riccati-like equations (4.12)-(4.22), which is proved by using a technique of control shifting. To the best of our knowledge, this result seems to be the first one of equivalent characterization on the convexity of the cost functional of LQ problem.

Based on this, necessary and sufficient conditions on the solvability of Problem (LQ) for a fixed initial pair is presented.

• •

Note that the constrained Riccati-like equations (5.47)-(5.59) and the LMEIs (5.15)-(5.31) are introduced for the first time, to the best of our knowledge. Furthermore, from a solution of the LMEIs, an explicit procedure is presented to construct a solution of the constrained Riccati-like equations. Such a procedure is potentially useful to study the algebraic Riccati-like equations that we will encounter in the infinite-horizon version of Problem (LQ).

It is worth mentioning that there are linear equations in the set of Riccati-like equations and the LMEIs contain equality constraints. Note that such new feature do not appear in deterministic LQ problems (with time delay) and standard stochastic LQ problems.

In [44], stochastic LQ problems with multiplicative noises and input delay were investigated, whose cost weighting matrices are assumed to be nonnegative definite. This paper is of general indefinite case, and thus, differs substaintially from [44]. In the context of this paper, it is proved in [44] that (ii) and (v) of Theorem 5.11 are equivalent for the nonnegative-definite case, which is the main result of the finite-horizon LQ problem in [44]. Note that in [44], the case with a fixed initial pair and the case with all the initial pairs are not differentiated, and no LMEIs are mentioned. Hence, the results of this paper are broader than those of the finite-horizon LQ problem of [44]. Furthermore, the transmission delay is studied in this paper, which is different from the input delay [44]; this is why the Riccati-like equations of this paper are divided into several pieces.

The rest of this paper is organized as follows. Section 2 and Section 3 give the problem formulation and an abstract consideration. In Section 4 and Section 5, the case with a fixed initial pair and the case with all the initial pairs are investigated, respectively. Section 6 gives an example, and some concluding remarks are given in Section 7.

2 Problem formulation

Consider the following controlled stochastic difference equation (S$\Delta$E, for short)

[TABLE]

where $A_{k},C_{k}\in\mathbb{R}^{n\times n}$, $B_{k},D_{k}\in\mathbb{R}^{n\times m}$ are deterministic matrices. The noise $\{w_{k},k\in\mathbb{T}\}$ is assumed to be a martingale difference sequence defined on a probability space $(\Omega,\mathcal{F},P)$ with

[TABLE]

Here, $\mathbb{E}_{k+1}$ is the conditional mathematical expectation $\mathbb{E}[\,\cdot\,|\mathcal{F}_{k+1}]$ with respect to $\mathcal{F}_{k+1}=\sigma\{w_{l},l=0,1,\cdots,k\}$, and $\mathcal{F}_{0}$ is understood as $\{\emptyset,\Omega\}$. Introduce the following cost functional associated with (2.3)

[TABLE]

where $Q_{k},R_{k},k\in\mathbb{T}_{t}$, $G$ are deterministic symmetric matrices of appropriate dimensions. Note, here, that we do not pose any definiteness constraints on the cost weighting matrices.

This paper is concerned with the case with transmission delay. For such kinds of time delays and the related measurement delays, find [3] [4] [8] [13] [14] [17] [21] [22] [24] [29] [33] [35] [39] [46] [47] in Introduction for their motivations and applications. Assume in this paper that there is a $d$-step time delay in the transmission/measurement channel ($d\geq 2$). Due to this, for $k\in\{t,...t+d\}$ no new information is available and the controller’s decision information set remains $\mathcal{F}_{t}$; and for $k\in\mathbb{T}_{t+d}=\{t+d,...,N-1\}$ the information set should be $\mathcal{F}_{k-d}$. In this paper, we select

[TABLE]

as the admissible control set, where

[TABLE]

and

[TABLE]

with

[TABLE]

Therefore, for any $(u_{t},...,u_{N-1})=u\in\mathcal{U}_{ad}^{t}$, $u_{k}$ is $\mathcal{F}_{t}$-measurable if $k\in\{t,...t+d\}$, and $u_{k}$ is $\mathcal{F}_{k-d}$-measurable if $k\in\mathbb{T}_{t}^{-d}$; this reflects the property of causality.

The following optimal control problem will be studied in this paper.

Problem (LQ). For a time-state initial pair $(t,x)\in\mathbb{T}\times l^{2}_{\mathcal{F}}(t;\mathbb{R}^{n})$, find a $\bar{u}\in\mathcal{U}_{ad}^{t}$ such that

[TABLE]

Remark 2.1**.**

Noting that the initial pair $(t,x)$ is specialized, hereafter the above problem will be called as Problem (LQ) for the initial pair $(t,x)$. Furthermore, any $\bar{u}$ satisfying (2.9) is called an optimal control of Problem (LQ) for the initial pair $(t,x)$.

Definition 2.2**.**

Problem (LQ) is said to be (uniquely) solvable at $(t,x)$ if there exists a (unique) $\bar{u}\in\mathcal{U}^{t}_{ad}$ such that (2.9) holds.

In what follows, we shall review some knowledge on matrix. Recall the pseudo-inverse of a matrix. By [28], for a given matrix $M\in\mathbb{R}^{n\times m}$, there exists a unique matrix in $\mathbb{R}^{m\times n}$ denoted by $M^{\dagger}$ such that

[TABLE]

This $M^{\dagger}$ is called the Moore-Penrose inverse of $M$. The following lemma is from [2].

Lemma 2.3**.**

Let matrices $L$, $M$ and $N$ be given with appropriate size. Then, $LXM=N$ has a solution $X$ if and only if $LL^{\dagger}NMM^{\dagger}=N$. Moreover, the solution of $LXM=N$ can be expressed as $X=L^{\dagger}NM^{\dagger}+Y-L^{\dagger}LYMM^{\dagger}$, where $Y$ is a matrix with appropriate size.

If $M=I$ in Lemma 2.3, then $LL^{\dagger}N=N$ is equivalent to $\mbox{Ran}(N)\subset\mbox{Ran}(L)$. Here, $\mbox{Ran}(N)$ is the range of $N$. The following is the so-called extended Schur’s lemma.

Lemma 2.4**.**

Let $S=S^{T}\in\mathbb{R}^{n\times n},W=W^{T}\in\mathbb{R}^{m\times m},H\in\mathbb{R}^{m\times n}$. Then

[TABLE]

if and only if

[TABLE]

3 An abstract consideration

For the completeness of theory, in this section, we convert Problem (LQ) to a quadratic optimization problem in Hilbert space, based on which some necessary conditions and sufficient conditions are given on the solvability of Problem (LQ). This part of work is a discrete-time version (with state transmission delay) of the results in [43], which will give us an overall perspective of Problem (LQ) and will motivate the analysis of the following sections.

To begin with, for $k,l\in\mathbb{T}_{t}$, let

[TABLE]

From (2.3), we have

[TABLE]

For any $x\in l^{2}_{\mathcal{F}}(t;\mathbb{R}^{n})$ and $u\in\mathcal{U}^{t}_{ad}$, define the following operators

[TABLE]

Hence,

[TABLE]

and

[TABLE]

It is not hard to see that the operators

[TABLE]

are all bounded and linear. Notice that the spaces in (3.8) are all Hilbert spaces. Therefore, the corresponding adjoint operators uniquely exist. For $\eta\in l^{2}_{\mathcal{F}}(N;\mathbb{R}^{n})$ and $\xi\in l^{2}_{\mathcal{F}}({{\mathbb{T}}}_{t};\mathbb{R}^{n})$, introduce the following BS$\Delta$E

[TABLE]

Proposition 3.1**.**

Let $V^{0}$ be the solution of (3.11) with $\eta=0$ and $V^{00}$ be the solution of (3.11) with $\xi=0$. Then the adjoint operators $\Gamma^{t*}$, $L^{t*}$, $\hat{\Gamma}^{t*}$ and $\hat{L}^{t*}$are given, respectively, by

[TABLE]

and

[TABLE]

In (3.13), (3.15), $\mathbb{E}_{k-d}$ is understood as $\mathbb{E}_{t}$ if $k\in\{t,...,t+d-1\}$ (i.e., $k-d<t$).

Proof. From (3.11) and by substituting $X_{k+1}$, we have

[TABLE]

Letting $\eta=0,u=0$ in (3), from (3.4) we have

[TABLE]

which implies (3.12). Letting $x=0,\eta=0$ in (3), the following equation holds

[TABLE]

Hence, the adjoint operator $L^{t*}$ of $L$ is given by (3.13).

Letting $\xi=0,u=0$ in (3), we have

[TABLE]

Then, the adjoint operator $\hat{\Gamma}^{t*}$ of $\hat{\Gamma}^{t}$ is given by (3.14). Furthermore, letting $\xi=0,x=0$ in (3), it holds that

[TABLE]

We therefore have (3.15).

We further use the convention

[TABLE]

Then, the cost functional $J(t,x;u)$ has the following form

[TABLE]

with

[TABLE]

In (3), the inner products are for different Hilbert spaces.

Based on above preparations, we have the following result.

Proposition 3.2**.**

The following statements hold.

(i) Let $u,v\in\mathcal{U}_{ad}^{t}$ and $\lambda\in\mathbb{R}$. Then

[TABLE]

(ii) Problem (LQ) is (uniquely) solvable at $(t,x)$ if and only if $\Theta^{t}_{1}\geq 0$ and there exists a (unique) $\bar{u}\in\mathcal{U}_{ad}^{t}$ such that

[TABLE]

(iii) If $\Theta^{t}_{1}>aI$ for some $a>0$, then $J(t,x;u)$ admits a unique minimizer $\bar{u}$

[TABLE]

In addition, if

[TABLE]

then, $\Theta^{t}_{1}>aI$ for some $a>0$.

Proof. (i) follows from (3), which implies (ii) and (iii).

Some calculations show

[TABLE]

and

[TABLE]

where $V^{1},V^{2}$ are given by

[TABLE]

and

[TABLE]

Hence, we have the following results.

Corollary 3.3**.**

Let $u,{v}\in\mathcal{U}_{ad}^{t}$ and $\lambda\in\mathbb{R}$. Then,

[TABLE]

where

[TABLE]

with $X$ is given in (2.3).

Proof. From Proposition 3.2, we need only to derive the expression of $\Theta_{1}^{t}u+\Theta_{2}^{t}x$. In fact, from (3.21)-(3.28) we have

[TABLE]

Noting (3.4) and (3.5), we will have (3.29).

4 Problem (LQ) for a fixed time-state initial pair

In this section, we will study Problem (LQ) for the fixed initial pair $(t,x)$, and the general case of Problem (LQ) for all the initial pairs will be introduced and studied in the next section. Throughout this section, Problem (LQ) for the fixed initial pair $(t,x)$ will be simply denoted as Problem (LQ)tx.

Throughout this paper, $\mathbb{E}_{k-d}$ is understood as $\mathbb{E}_{t}$ if $k\in\{t,...,t+d-1\}$ (i.e., $k-d<t$). From Proposition 3.2 and Corollary 3.3, the following theorem is straightforward.

Theorem 4.1**.**

The following statements are equivalent.

(i) Problem (LQ)tx is solvable.

(ii) The following assertions hold.

• • a)

There exists a $u^{{t,x,*}}\in\mathcal{U}_{ad}^{t}$ such that the stationary condition

[TABLE]

is satisfied, where $Z^{t,x,*}$ is the backward state of the following FBS$\Delta$E

[TABLE]

• b)

The convexity condition

[TABLE]

holds.

Under any of the above conditions, $u^{{t,x,*}}$ in (ii) is an optimal control of Problem (LQ)tx.

4.1 Stationary condition

By the stationary condition (4.1) and a backward procedure of calculations, we can get the following discrete-time Riccati-like equations

[TABLE]

[TABLE]

and

[TABLE]

where

[TABLE]

and

[TABLE]

Furthermore, the backward state of FBS$\Delta$E (4.5) can be expressed via the forward state and the solution of (4.12)-(4.22). Due to the $d$-step-lagged information structure, the Riccati-like equations are much more complicated than the standard discrete-time Riccati equation. In fact, we have the following equivalent characterization of the stationary condition.

Theorem 4.2**.**

The following statements are equivalent.

(i) The stationary condition of (4.1) is satisfied for some $u^{t,x,*}\in\mathcal{U}_{ad}^{t}$.

(ii) The following condition

[TABLE]

is satisfied, where $W_{k},H_{k},k\in\mathbb{T}_{t}$, are given in (4.25) and (4.28), and $X^{t,x,*}$ is given by the forward S$\Delta$E of

[TABLE]

with

[TABLE]

Furthermore, the backward state $Z^{t,x,*}$ of (4.33) has the following expression

[TABLE]

where $P^{(i)},i=0,...,d$, are given in (4.12)-(4.22).

Proof. See Appendix A.

Remark 4.3**.**

If $x$ in (4.33) is 0, then $X^{t,0,*}_{k}=0,k\in\mathbb{T}_{t}$. In this case, the condition (4.29) is naturally satisfied.

Remark 4.4**.**

From the proof of Theorem 4.2, we know that the key technique is to decouple the FBS$\Delta$E (4.5) by virtue of (4.1), i.e., find the expression (4.37). Due to the delayed information structure, at $k\in\{t,...,t+d\}$ the decision information set remains $\mathcal{F}_{t}$. For $k\in\{t,...,t+d-1\}$, $Z^{t,x,*}_{k}$ is a linear function of $X^{t,x,*}_{k},\mathbb{E}_{k-1}X^{t,x,*}_{k},\cdots,P^{(k-t)}_{k}\mathbb{E}_{t}X_{k}^{t,x,*}$, which differs from the case of $k\in\mathbb{T}_{t+d}$. This is why the Riccati-like equations are divided into several pieces (4.12)-(4.22). Letting $k=t,t+1$, then $k-t-1$ in (4.17) will be 0 and 1. Hence, (4.22) is not a special form of (4.17).

Remark 4.5**.**

Substituting (4.34) into the forward S$\Delta$E of (4.33), we have

[TABLE]

Taking conditional expectations in both sides of all the equations of (4.40), we have

[TABLE]

For $k\in\mathbb{T}_{t+d}$ and by successive iterations, it holds

[TABLE]

which is eventually a linear function of $X^{t,x,*}_{k+1-d},...,X^{t,x,*}_{t}$. Similar expressions can be derived for the case of $k\in\{t,...,t+d-1\}$. Combining this and (4.40), we can get all the values of $\mathbb{E}_{k-d}X_{k}^{t,x,*},k\in\mathbb{T}_{t}$. Hence, the control (4.34) can be easily implemented.

The following result shows that the solution of (4.12)-(4.22) can be calculated through a set of Riccati-like equations.

Proposition 4.6**.**

Let

[TABLE]

where

[TABLE]

Then for (4.12)-(4.22) it holds that

[TABLE]

Proof. $P_{k}^{(i)}=\bar{P}_{k}^{(i)}$ follows from their expressions for the case with $k\in\mathbb{T}_{t+d},i=0,...,d$ and the case with $k\in\{t+2,...,t+d-1\},i=0,...,k-t-1$. For $k=t+d-1$ and $i=d-1$,

[TABLE]

Furthermore, we have

[TABLE]

where we have used the properties

[TABLE]

and $W_{t+d-2}=\bar{W}_{t+d-2}$. By induction, we can achieve the conclusion.

Remark 4.7**.**

(4.47) with $\bar{W}_{k}>0,k\in\mathbb{T}_{t}$, is first introduced in [44], which characterizes the unique solvability of stochastic LQ problem with input delay. We here will investigate Problem (LQ) with information transmission delay, and intend seeking more general conditions to ensure the solvability of Problem (LQ) for the case with a fixed initial pair and the case with all the initial pairs.

4.2 Convexity

We now study the convexity condition. In what follows, the functional $u\mapsto J(t,x;u)$ is called convex if (4.6) holds.

Lemma 4.8**.**

For any $u\in\mathcal{U}_{ad}^{t}$, it holds that

[TABLE]

with $X^{0}$ given by

[TABLE]

Proof. See Appendix B.

As $W_{k},k\in\mathbb{T}_{t},$ are symmetric, there exist orthogonal matrices $F_{k},k\in\mathbb{T}_{t}$, such that

[TABLE]

In the above, $\Sigma_{k}\in\mathbb{R}^{r_{k}\times r_{k}}$ is a diagonal matrix with $r_{k}$ being the rank of $W_{k}$, whose diagonal elements are the nonzero eigenvalues of $W_{k}$. Hence, we have

[TABLE]

Moreover, $F_{k}$ can be decomposed as $F_{k}^{T}=[(F^{(1)}_{k})^{T}~{}~{}(F^{(2)}_{k})^{T}]$, where the lines of $F^{(2)}_{k}\in\mathbb{R}^{(m-r_{k})\times m}$ form a basis of $\mbox{Ker}(W_{k})$ (the kernel of $W_{k}$). Let

[TABLE]

Hence, (7.10) becomes to

[TABLE]

Note that the space spanned by lines of $F_{k}^{(1)}$ is $\mbox{Ran}(W_{k})$, the range of $W_{k}$. Let $\mathcal{U}_{ad}^{t}(\mbox{Ker})$ be a subset of $\mathcal{U}_{ad}^{t}$ such that for any $u\in\mathcal{U}_{ad}^{t}(\mbox{Ker})$, $u_{k}\in\mbox{Ker}(W_{k}),k\in\mathbb{T}_{t}$. Similarly, $\mathcal{U}_{ad}^{t}(\mbox{Ran})$ is defined as a subset of $\mathcal{U}_{ad}^{t}$ such that for any $u\in\mathcal{U}_{ad}^{t}(\mbox{Ran})$, $u_{k}\in\mbox{Ran}(W_{k}),k\in\mathbb{T}_{t}$.

By the above preparations, we have the following theorem, which gives necessary and sufficient conditions on the convexity of $u\mapsto J(t,x;u)$. To the best of our knowledge, it seems to be the first result to equivalently characterize the convexity of LQ problem.

Theorem 4.9**.**

The following statements are equivalent.

(i) $u\mapsto J(t,x;u)$ is convex.

(ii) The following assertions hold.

• • a)

The solution of Riccati-like equation set (4.12)-(4.22) has the property $W_{k}\geq 0,k\in\mathbb{T}_{t}$.

• b)

For any $u\in\mathcal{U}_{ad}^{t}$, the condition

[TABLE]

is satisfied, where $X^{0,u}$ is given by

[TABLE]

with

[TABLE]

Proof. $(i)\Rightarrow(ii)$. As $u\mapsto J(t,x;u)$ is convex, from (4.60) we have

[TABLE]

where $u_{k}^{(1)}$ and $u_{k}^{(2)}$ are the projections of $u_{k}$ onto $\mbox{Ran}(W_{k})$ and $\mbox{Ker}(W_{k})$, respectively. Then, it holds that

[TABLE]

Introduce a set

[TABLE]

Note that for $k\in\mathbb{T}_{t}$, the lines, $\alpha_{k}^{1},...,\alpha_{k}^{r_{k}}$, of $F^{(1)}_{k}$ form a basis of $W_{k}$. For $u=\{u_{t},...,u_{N-1}\}\in\mathcal{U}^{t}_{ad}(\mbox{Ran})$, for each $k\in\mathbb{T}_{t}$ there exist $\lambda_{k}^{1},...,\lambda_{k}^{r_{k}}\in\mathbb{R}$ such that $u_{k}=\sum_{i=1}^{r_{k}}\lambda_{k}^{i}(\alpha_{k}^{i})^{T}$. Then,

[TABLE]

For $k\in{T}_{t+d+1}=\{t+d+1,...,N-1\}$, $u_{k}$ is $\mathcal{F}_{k-d}$-measurable and $\mathbb{E}|u_{k}|^{2}<\infty$. This implies that $\lambda_{k}$ is $\mathcal{F}_{k-d}$-measurable and $\mathbb{E}|\lambda_{k}|^{2}<\infty$. Similar result holds for the case of $k\in\{t,...,t+d\}$. Therefore,

[TABLE]

Here, $L^{2}_{\mathcal{F}}(t;\mathbb{R}^{r_{k}})$ and $L^{2}_{\mathcal{F}}(k-d;\mathbb{R}^{r_{k}}),k\in\mathbb{T}_{t+d}$, are similarly defined as (2.7). Therefore,

[TABLE]

Introduce a bounded linear operator from ${\mathcal{U}}^{t}_{ad}(\mbox{Ran})$ to $\widetilde{\mathcal{U}}^{t}_{ad}(\mbox{Ran})$

[TABLE]

Here, $X^{0}$ is the solution of (4.56). We now prove that $\mathcal{L}$ is a surjection. In fact, for any $\theta\in\widetilde{\mathcal{U}}^{t}_{ad}(\mbox{Ran})$, let

[TABLE]

and

[TABLE]

Note that $u$ given in (4.71) is in $\mathcal{U}_{ad}^{t}(\mbox{Ran})$. As $F_{k}^{(1)}(F_{k}^{(1)})^{T}=I_{r_{k}}$, from (4.71) we have

[TABLE]

Hence, $\mathcal{L}$ is a surjection defined from ${\mathcal{U}}^{t}_{ad}(\mbox{Ran})$ to $\widetilde{\mathcal{U}}^{t}_{ad}(\mbox{Ran})$. From this, (4.2) and the proof by contradiction, we have $\Sigma_{k}>0,k\in\mathbb{T}_{t}$. This further implies $W_{k}\geq 0,k\in\mathbb{T}_{t}$. Then, a) is proved.

We now prove b). Note that (4.65) equals to

[TABLE]

For any $u\in\mathcal{U}_{ad}^{t}$, there exist $u^{(1)}\in\mathcal{U}_{ad}^{t}(\mbox{Ran})$ and $u^{(2)}\in\mathcal{U}_{ad}^{t}(\mbox{Ker})$ such that $u=u^{(1)}+u^{(2)}$, i.e., $u_{k}=u_{k}^{(1)}+u_{k}^{(2)},k\in\mathbb{T}_{t}$. From (4.60), for any $u\in\mathcal{U}^{t}_{ad}$ we have

[TABLE]

In the above, we have used the properties $F^{(i)}_{k}u_{k}=F^{(i)}_{k}u^{(i)}_{k},i=1,2$, and $F^{(2)}_{k}(F^{(1)}_{k})^{T}=0$. From (4.72), we must have

[TABLE]

Otherwise, we can select some $u$ such that $J(t,0;v^{u})<0$. In fact, assume there exist $k_{1}\in\mathbb{T}_{t}$ and $\widehat{u}\in\mathcal{U}_{ad}^{t}$ such that

[TABLE]

Denote

[TABLE]

Introduce a new control

[TABLE]

which is in $\mathcal{U}^{t}_{ad}(\mbox{Ker})$. Then, we have

[TABLE]

This contradicts the convexity of $u\mapsto J(t,x;u)$. Hence, we have (4.73). By multiplying $F_{k}^{(2)}$ and noting $F^{(2)}_{k}(F^{(2)}_{k})^{T}=I_{m-r_{k}}$, it holds that

[TABLE]

which is equivalent to (4.61).

$(ii)\Rightarrow(i)$. From the condition (ii), (4.60), (4.61) and (4.75), we have for any $u\in\mathcal{U}_{ad}^{t}$

[TABLE]

We now show

[TABLE]

where $v^{u}$ is given by (4.65). For any ${\widetilde{v}}\in\mathcal{U}_{ad}^{t}$, let

[TABLE]

where

[TABLE]

We then have from (4.65) (4.78) that $v^{u}={\widetilde{v}}$. Hence, (4.77) holds, which together with (4.76) implies

[TABLE]

This completes the proof.

In the proof of $(ii)\Rightarrow(i)$ of Theorem 4.9, we have used a simple technique of control shifting ($u\mapsto v^{u}$). To make it more clear, we state the following lemma, whose proof is omitted here. Firstly, introduce a set; let $L^{2}(\mathbb{T}_{t};\mathbb{R}^{m\times n})$ be a set of $\mathbb{R}^{m\times n}$-valued deterministic processes such that for any $\nu=\{\nu_{k},k\in\mathbb{T}_{t}\}\in L^{2}(\mathbb{T}_{t};\mathbb{R}^{m\times n})$, $\sum_{k=t}^{N-1}|\nu_{k}|^{2}<\infty$.

Proposition 4.10**.**

For any $\Phi\in L^{2}(\mathbb{T}_{t};\mathbb{R}^{m\times n})$, the following assertions hold.

(i) The property

[TABLE]

is satisfied, where ${u}-\Phi\mathbb{E}_{\cdot-d}X$ is the control $\{{u}_{k}-\Phi_{k}\mathbb{E}_{k-d}X_{k},~{}~{}k\in\mathbb{T}_{t}\}$ with

[TABLE]

(ii) it holds that

[TABLE]

Furthermore, by some simple calculations, we can show that the convexity of $u\mapsto J(t,x;u)$ has a semi-global property in the sense of the following result.

Proposition 4.11**.**

If $u\mapsto J(t,x;u)$ is convex, then for any $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$ and $u\in\mathcal{U}_{ad}^{t}$, $u|_{\mathbb{T}_{k}}\mapsto J(k,\xi;u|_{\mathbb{T}_{k}})$ is also convex. Here, $\mathbb{T}_{k}=\{k,...,N-1\}$, and $u|_{\mathbb{T}_{k}}$ is the restriction of $u$ on $\mathbb{T}_{k}$.

4.3 The solvability of Problem (LQ)tx

Noting (4.34) and (4.33), we have

[TABLE]

with property (4.29). Letting $X^{x,u}=X^{t,x,*}+X^{0,u}$ and from (4.64) (4.84), we have

[TABLE]

From Theorem 4.1, Theorem 4.2 and Theorem 4.9, we then have the following necessary and sufficient conditions on the existence of optimal control of Problem (LQ)tx.

Theorem 4.12**.**

The following statements are equivalent.

(i) Problem (LQ)tx is solvable.

(ii) The following assertions hold

• • a)

The solution of Riccati-like equation set (4.12)-(4.22) has the property $W_{k}\geq 0,k\in\mathbb{T}_{t}$.

• b)

For any $u\in\mathcal{U}_{ad}^{t}$, the condition

[TABLE]

is satisfied, where $X^{x,u}$ is the solution of (4.88).

Under any of the above conditions, the following control

[TABLE]

is an optimal control of Problem (LQ)tx, where $X^{t,x,*}$ is given by (4.84).

Proof. The equivalence between (i) and (ii) follows from the construction of $X^{x,u}$. From Proposition 4.10, we have

[TABLE]

with $v^{u}=\{v^{u}_{k}=u_{k}-W_{k}^{\dagger}H_{k}\mathbb{E}_{k-d}X^{x,u}_{k},k\in\mathbb{T}_{t}\}$. Under any of (i) (ii) and similar to (7.10), we have

[TABLE]

where for $u=0$ the equality holds. In the above, we have used the property (4.89), which is equivalent to

[TABLE]

By (4.91) and (4.92), we then achieve the conclusion.

Introduce a set

[TABLE]

Theorem 4.13**.**

$\mathcal{I}_{t}$* is either empty or a linear subspace of $\mbox{Ker}[(I-W_{t}W_{t}^{\dagger})H_{t}]$.*

Proof. Letting $u=0$ in (4.88), we have $X^{x,0}=X^{t,x,*}$, which is given in (4.84). For $x\in\mathcal{I}_{t}\neq\varnothing$, $x$ will be in $\mbox{Ker}[(I-W_{t}W_{t}^{\dagger})H_{t}]$. Then, for $x,x^{\prime}\in\mathcal{I}_{t},\alpha,\beta\in\mathbb{R}$, we have

[TABLE]

Hence, $\alpha X^{x,0}+\beta X^{x^{\prime},0}=X^{\alpha x+\beta x^{\prime},0}$, which further implies

[TABLE]

Combining with the convexity, we know that Problem (LQ) is solvable at $(t,\alpha x+\beta x^{\prime})$.

To end this subsection, a sufficient condition is presented to ensure (4.89).

Theorem 4.14**.**

If $\mbox{Ran}(H_{k})\subset\mbox{Ran}(W_{k})$ (i.e., $W_{k}W_{k}^{\dagger}H_{k}=H_{k}$), $k\in\mathbb{T}_{t}$, then the condition (4.89) is satisfied.

Proof. The proof is straightforward and hence, omitted here.

Combining the condition in Theorem 4.14 with a) of Theorem 4.12, we can obtain in the next section much neater results of Problem (LQ) (for all the initial pairs).

4.4 The delay-free case

Let us revisit the standard discrete-time stochastic LQ problem without time delay.

Problem (LQ)${}^{df}_{tx}$. For the initial pair $(t,x)\in\mathbb{T}\times\mathbb{R}^{n}$, find a $\bar{u}\in\mathcal{U}_{ad}^{t}$ such that

[TABLE]

In (4.94), $J(t,x;u)$ is given in (2.5) and is subject to (2.3), and

[TABLE]

Introduce the discrete-time Riccati equation

[TABLE]

where

[TABLE]

Let $d=0$ in Theorem 4.12, we have the following result.

Theorem 4.15**.**

The following statements are equivalent.

(i) Problem (LQ)${}^{df}_{tx}$ is solvable.

(ii) The following assertions hold

• • a)

The solution of Riccati equation (4.97) has the property $W_{k}\geq 0,k\in\mathbb{T}_{t}$.

• b)

For any $u\in L^{2}_{\mathcal{F}}(\mathbb{T}_{t};\mathbb{R}^{m})$, the condition

[TABLE]

is satisfied, where $X^{x,u}$ is the solution of the following S$\Delta$E

[TABLE]

with

[TABLE]

Firstly, let us take some observation. Let $V_{k}=(I-W_{k}^{\dagger}W_{k})H_{k},k\in\mathbb{T}_{t}$. Then, the condition (4.99) is equivalent to

[TABLE]

Hence, at $k$, the attainable set of the system (4.102) is a subset of $\mbox{Ker}(V_{k})$. Similarly, (4.89) is relating to the property of the attainable set of system (4.88). To get conditions of (4.89) (4.99) that are easier to validated, we should in the future to study the attainable set of (4.102) and (4.88), which is further related to the controllability of linear S$\Delta$Es.

Letting the initial pair $(t,x)$ vary in the product space $\mathbb{T}\times\mathbb{R}^{n}$, we get a family of LQ problems; from Theorem 4.15, we have an equivalent characterization of the solvability of this family of LQ problems.

Proposition 4.16**.**

The following statements are equivalent.

(i) For any $(t,x)\in\mathbb{T}\times\mathbb{R}^{n}$, Problem (LQ)${}^{df}_{tx}$ is solvable.

(ii) The constrained Riccati equation

[TABLE]

is solvable in the sense $W_{k}^{\dagger}W_{k}H_{k}=H_{k},~{}W_{k}\geq 0,~{}k\in\mathbb{T}_{t}$, where

[TABLE]

Proof. For any $t\in\mathbb{T}$ and letting $k=t$ in (4.99), we have $H_{t}x\in\mbox{Ran}(W_{t})$, which holds for any $x\in\mathbb{R}^{n}$; equivalently, we have $\mbox{Ran}(H_{t})\subset\mbox{Ran}(W_{t})$ by considering the cases $x=e_{i},i=1,...,n$. Here, $e_{i}$ is the $n$-dimensional vector, whose $i$-th entry is 1 and other entries are all 0. Combining this fact and Theorem 4.15, we then achieve the result.

Remark 4.17**.**

Proposition 4.16 is a main result of [2], which presents a necessary and sufficient condition on the solvability of a family of LQ problems, that is,

{Problem (LQ)${}^{df}_{tx}$* is solvable, for any $(t,x)\in\mathbb{T}\times\mathbb{R}^{n}$}$\Leftrightarrow$ (4.107) is solvable.*

In contrast, Theorem 4.15 just characterizes the solvability of Problem (LQ)${}^{df}_{tx}$. The proof of Proposition 4.16 shows that Theorem 4.15 implies Proposition 4.16. However, the equivalence between (i) and (ii) of Theorem 4.15 cannot be proved by virtue of Proposition 4.16. Hence, Theorem 4.15 is a new result even for standard LQ problems; a key step to access to Theorem 4.15 is that we have obtained (for the first time) an equivalent characterization of the convexity of the cost functional.

5 Problem (LQ) for all the time-state initial pairs

5.1 The solvability of Problem (LQ)

In this section, we will study Problem (LQ) for all the initial pairs. To begin with, we introduce versions of Problem (LQ) (for the initial pair $(t,x)$). If $k\in\{t,...,t+d-1\}$, let

[TABLE]

if $k\in\mathbb{T}_{t+d}$, let

[TABLE]

In (5.2), $l^{2}_{\mathcal{F}}(\mathbb{T}^{-d}_{k-d};\mathbb{R}^{m})$ is a set of $\mathbb{R}^{m}$-valued processes with $\mathbb{T}_{k-d}^{-d}=\{k-d,...,N-1-d\}$ such that for any its element $\nu=\{\nu_{\ell},\ell\in\mathbb{T}_{k-d}^{-d}\}$, $\nu_{\ell}$ is $\mathcal{F}_{\ell}$-measurable and $\sum_{\ell=k-d}^{N-1-d}\mathbb{E}|\nu_{\ell}|^{2}<\infty$.

Starting from the initial pair $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$, (2.3) (2.5) become to

[TABLE]

and

[TABLE]

Problem (LQ) for the initial pair $(k,\xi)$ is referred to as the case that minimizes (5.6) over $\mathcal{U}^{k}_{ad}$ (subject to (5.5)).

Definition 5.1**.**

(i) Problem (LQ) is said to be finite at $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$, if

[TABLE]

Problem (LQ) is said to be finite if (5.7) holds for any initial pair $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$.

(ii) Problem (LQ) is said to be (uniquely) solvable at $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$ if there exists a (unique) $\bar{u}\in\mathcal{U}^{k}_{ad}$ such that

[TABLE]

In this case, $\bar{u}$ is called an optimal control of Problem (LQ) for the initial pair $(k,\xi)$. Problem (LQ) is said to be (uniquely) solvable if it is solvable at any initial pair $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$.

To study the finiteness of Problem (LQ), introduce the following coupled LMEIs (5.15)-(5.31)

[TABLE]

[TABLE]

and

[TABLE]

where

[TABLE]

and

[TABLE]

If exists, the solution of (5.15)-(5.31) is denoted as $(P^{(0)},...,P^{(d)})$. Let

[TABLE]

Based on the solution of (5.15)-(5.31), we have the following lemma, whose proof is similar to that of Lemma 4.8 and is omitted here.

Lemma 5.2**.**

Let $(P^{(0)},...,P^{(d)})\in\mathcal{M}\neq\varnothing$. Then the following statements hold.

(i) For $k\in\mathbb{T}_{t+d}$, it holds that

[TABLE]

(ii) For $k\in\{t+1,...,t+d-1\}$, it holds that

[TABLE]

(iii) It holds that

[TABLE]

Remark 5.3**.**

The LMEIs (5.15)-(5.31) are such constructed that the inequalities of Lemma 5.2 are satisfied. In this case, Problem (LQ) will be finite. Note that the LMEIs contain equality constraints; such new feature does not appear in deterministic LQ problems (with time delay) and standard stochastic LQ problems.

Based on above preparations, we have the following theorem, which gives several equivalent characterizations on the solvability of Problem (LQ).

Theorem 5.4**.**

The following statements are equivalent.

(i) Problem (LQ) is finite.

(ii) Problem (LQ) is solvable.

(iii) The solution of (4.12)-(4.22) has the property $W_{k}W_{k}^{\dagger}H_{k}=H_{k},W_{k}\geq 0,k\in\mathbb{T}_{t}$, namely, the constrained Riccati-like equation set

[TABLE]

[TABLE]

and

[TABLE]

are solvable in the sense

[TABLE]

(iv) $\mathcal{M}$ defined in (5.34) is nonempty.

Under any of the above conditions, an optimal control of Problem (LQ) for the initial pair $(k,\xi)$ is given by

[TABLE]

with

[TABLE]

Furthermore, the corresponding optimal value is

[TABLE]

Proof. See Appendix C.

Corollary 5.5**.**

Let $Q_{k}\geq 0,R_{k}\geq 0,G\geq 0,k\in\mathbb{T}_{t}$. Then, Problem (LQ) is solvable, and the corresponding Riccati-like equations (5.47)-(5.59) are solvable.

Proof. In this case, Problem (LQ) is finite, and the conclusion follows from Theorem 5.4.

5.2 From the LMEIs to the Riccati-like equations

——construct the solution of (5.47)-(5.59)

In this subsection, a procedure is presented to construct the solution of Riccati-like equation set (5.47)-(5.59) from an element $(\widetilde{P}^{(0)},...\widetilde{P}^{(d)})$ of $\mathcal{M}$ (i.e., a solution of the LMEIs (5.15)-(5.31)). To do so, we introduce an auxiliary LQ problem.

Specifically, introduce the following weighting matrices

[TABLE]

Furthermore, for each $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$, let $X$ be the solution of (5.5) and introduce the cost functional $\widetilde{J}(k,\xi;u)$ according to three different situations. Case 1: $k\in\mathbb{T}_{t+d}$, let

[TABLE]

Case 2: $k\in\{t+1,...,t+d-1\}$, let

[TABLE]

Case 3: $k=t$, let

[TABLE]

Corresponding to the above cost functional (5.2)-(5.2), the system (5.5) and the admissible control set (5.1)-(5.2), we denote such an LQ problem as Problem (LQ)a for the initial pair $(k,\xi)$.

The cost functional $\widetilde{J}(k,\xi;u)$ is such constructed in (5.2)-(5.2) that it is finite from below. This is proved in the following proposition.

Proposition 5.6**.**

For any $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$, $\widetilde{J}(t,\xi;u)\geq 0$. Hence, Problem (LQ)a is finite.

Proof. For (5.2), we have

[TABLE]

The inequality above is due to the fact $(\widetilde{P}^{(0)},...,\widetilde{P}^{(d)})\in\mathcal{M}$. Similarly, we can prove other cases. Hence, $\widetilde{J}(k,\xi;u)\geq 0$ for any $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$.

Let us make some observations about $\widetilde{J}(t,\xi;u)$. By adding to and subtracting

[TABLE]

from $\widetilde{J}(t,\xi;u)$, we have

[TABLE]

In the above, $(U^{(0)},...,U^{(d)})$ is to be determined and

[TABLE]

and

[TABLE]

In fact, introduce the Riccati-like equation set

[TABLE]

[TABLE]

and

[TABLE]

with $\mathcal{W}_{k},\mathcal{H}_{k}$ being given by (5.83) (5.86); by analysis similar to (5.80), we then have the following result.

Lemma 5.7**.**

Let $(U^{(0)},...,U^{(d)})$ be the solution of (5.92)-(5.102). Then,

[TABLE]

where

[TABLE]

Based on what we have prepared above, we can construct a solution of (5.47)-(5.59) from $(\widetilde{P}^{(0)},...,\widetilde{P}^{(d)})\in\mathcal{M}$.

Theorem 5.8**.**

The following statements hold.

(i) The solution of Riccati-like equation set (5.92)-(5.102) has the property

[TABLE]

(ii) Let $P_{k}^{(i)}=\widetilde{P}_{k}^{(i)}+U_{k}^{(i)},k\in\mathbb{T}_{t},i=0,...,d$. Then, such a $(P^{(0)},...,P^{(d)})$ is a solution of the constrained Riccati-like equation set (5.47)-(5.59).

Proof. From Proposition 5.6, Problem (LQ)a is finite for any initial pair $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$; hence it is solvable. Combining Lemma 5.7 and the part of proving the equivalence between (i) and (iii) of Theorem 5.4, we must have (i) of this theorem. (ii) follows from some simple calculations.

Remark 5.9**.**

By Theorem 5.8, we can construct a solution of the constrained Riccati-like equation set from a solution of the LMEIs. This result is potentially useful to study the algebraic Riccati-like equations that we will encounter in the infinite-horizon version of Problems (LQ). For more about standard infinite-horizon stochastic LQ problems, we can refer to, for example, [5] [40].

5.3 The unique solvability of Problem (LQ)

In the following, we will study the uniform convexity of the cost functional, which is motivated by some results of [30]. The functional $u\mapsto J(t,x;u)$ is called uniformly convex if there exists a $\lambda>0$ such that for any $u\in\mathcal{U}_{ad}^{t}$

[TABLE]

From Proposition 3.2, Problem (LQ) will have a unique optimal control if $u\mapsto J(t,0;u)$ is uniformly convex.

Lemma 5.10**.**

For $\Phi\in L^{2}(\mathbb{T}_{t};\mathbb{R}^{m\times n})$ and (4.56) with $u\in\mathcal{U}_{ad}^{t}$, there exist $\gamma_{1},\gamma_{2}$ with property $0<\gamma_{2}<\gamma_{1}$ such that

[TABLE]

Proof. For $\Phi\in L^{2}(\mathbb{T}_{t};\mathbb{R}^{m\times n})$, define a bounded linear operator for $\mathcal{U}_{ad}^{t}$ to $\mathcal{U}_{ad}^{t}$

[TABLE]

where ${u}-\Phi\mathbb{E}_{\cdot-d}X^{0}$ is the control $\{{u}_{k}-\Phi_{k}\mathbb{E}_{k-d}X^{0}_{k},~{}~{}k\in\mathbb{T}_{t}\}$. Note that $\mathcal{O}u=0$ implies $u=0$, i.e., $\mathcal{O}$ is an injection. Let

[TABLE]

which is indeed a norm on $\mathcal{U}_{ad}^{t}$. Furthermore, for any given $u^{(n)}\in\mathcal{U}^{t}_{ad}$, we have when $n\mapsto\infty$

[TABLE]

Therefore, $p_{\Phi}(\,\cdot\,)$ is equivalent to the norm $||\cdot||$ on $\mathcal{U}_{ad}^{t}$. We then claim (5.105).

Theorem 5.11**.**

The following statements are equivalent.

(i) Problem (LQ) is uniquely solvable at the initial pair $(t,x)$.

(ii) Riccati-like equation set (5.47)-(5.59) is solvable, and $W_{k}>0,k\in\mathbb{T}_{t}$.

(iii) $u\mapsto J(t,x;u)$ is uniformly convex for $u\in\mathcal{U}^{t}_{ad}$.

(iv) For any $k\in\mathbb{T}_{t}$, $u\mapsto J(k,\xi;u)$ is uniformly convex for $u\in\mathcal{U}^{k}_{ad}$.

(v) Problem (LQ) is uniquely solvable at any initial pair $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$.

Under any of the above conditions, the optimal control of Problem (LQ) for the initial pair $(k,\xi)$ is given by

[TABLE]

with $X^{k,\xi,*}$ given by

[TABLE]

Proof. $(i)\Rightarrow(ii)$. This can be achieved by undating the proof of Theorem 4.2. Let $u^{t,x,*}$ be the unique optimal control of Problem (LQ) for the initial pair $(t,x)$. Noting (7.5) and that Theorem 4.1 is of necessary and sufficient conditions, we have

[TABLE]

As the optimal control uniquely exists, we must have that $W_{k}$ is nonsingular, $k\in\mathbb{T}_{t}$. Otherwise, any controls of the following form

[TABLE]

is also an optimal control, where

[TABLE]

Since $W_{k},k\in\mathbb{T}_{t}$, are all invertible, from (7.10) and (4.6) we have

[TABLE]

where $X^{0}$ is given in (4.56). Letting $\Phi_{k}=-W^{-1}_{k}H_{k}$, from Lemma 4.10 we know that the linear operator $\mathcal{O}$ defined in (5.106) is a surjection from $\mathcal{U}^{t}_{ad}$ to $\mathcal{U}^{t}_{ad}$. Noting that (5.111) holds for any $u\in\mathcal{U}^{t}_{ad}$, we must have

[TABLE]

which implies (ii).

$(ii)\Rightarrow(iii)$. Similarly to (5.111), it holds that

[TABLE]

From Lemma 5.10, we have

[TABLE]

where $\lambda_{min}>0$ denotes the minimal eigenvalue among all the eigenvalues of $W_{k},k\in\mathbb{T}_{t}$. Hence, $u\mapsto J(t,x;u)$ is uniformly convex.

(iii)$\Rightarrow$(iv). Let $u\mapsto J(t,x;u)$ be uniformly convex for $u\in\mathcal{U}^{t}_{ad}$. Now for any $u=(u_{k},...,u_{N-1})\in\mathcal{U}_{ad}^{k}$, let $v=(0,...,0,u_{k},...,u_{N-1})\in\mathcal{U}_{ad}^{t}$. Then, we have

[TABLE]

Hence, $u\mapsto J(k,\xi;u)$ is uniformly convex.

(iv)$\Rightarrow$(v). From Proposition 3.2, Problem (LQ) for the initial pair $(k,\xi)$ admits a unique optimal control.

(v)$\Rightarrow$(i). This is clear.

Under any of the above conditions, we have (5.107).

Remark 5.12**.**

The theorem above shows that Problem (LQ) is uniquely solvable at the initial pair $(t,x)$ if and only if Problem (LQ) is uniquely solvable at any initial pair $(k,\xi)\in\mathbb{T}_{t}\times\mathbb{R}^{n}$. This result links Section 4 with this section. Note, here, that the condition of uniform convexity plays a key role.

6 Example

In this section, we shall present an example to illustrate the theory derived above.

Example 6.1**.**

Consider a version of Problem (LQ) whose system matrices and weighting matrices are

[TABLE]

Let $N=4$ and $d=2$ in (2.3) in (2.6). Find the optimal control.

In this case, the constrained Riccati-like equation set (5.47)-(5.59) becomes to

[TABLE]

and

[TABLE]

By some calculations, we have

[TABLE]

which are positive definite. Hence, (6.14)-(6.20) are solvable. Furthermore, the unique optimal control is given by

[TABLE]

where $-W_{k}^{\dagger}H_{k},k=0,1,2,3$, are

[TABLE]

and $X^{0,x,*}$ is given by

[TABLE]

7 Conclusion

In this paper, an indefinite stochastic LQ problem with transmission delay and multiplicative noises is studied. Based on some abstract consideration, necessary and sufficient conditions are given, respectively, for the case with a fixed initial pair and the case with all the initial pairs. Further, a set of constrained discrete-time Riccati-like equations and a set of linear matrix equality-inequalities are introduced, which are used to characterize the existence of the delayed optimal control of Problem (LQ). Moreover, the unique solvability of the delayed optimal control is also fully characterized. For future research, the infinite-horizon stochastic LQ problem with input delay should be investigated.

Appendix

A. Proof of Theorem 4.2

$(i)\Rightarrow(ii)$. Let $u^{t,x,*}$ be an optimal control of Problem (LQ) for the initial pair $(t,x)$. Then, we now prove that (4.29) is satisfied with property (4.37). The following deduction is a variant of that in [44].

Firstly, let us begin with the cast $k=N-1$. Noting $Z^{t,x,*}_{N}=GX_{N}^{t,x,*}$, we have

[TABLE]

and

[TABLE]

Hence, (4.1) for $k=N-1$ reads as

[TABLE]

As there exists a $u^{t,x,*}$ satisfies (4.1), from Lemma 2.3 we know that (4.29) holds for $k=N-1$, and that $u^{t,x,*}_{N-1}$ can be selected as

[TABLE]

Furthermore,

[TABLE]

In view of $P^{(i)}_{N-1}=0,i=1,\cdots,d-1$, we have (4.37) for $k=N-1$.

Secondly, assume that for $k\in\{t+d,...,N-1\}$

[TABLE]

and

[TABLE]

Now we verify that these are also true for the case $k$. In fact, notice that

[TABLE]

and

[TABLE]

Then, (4.1) reads as

[TABLE]

This implies by Lemma 2.3 that (4.29) holds for $k$ and that $u^{t,x,*}_{k}$ can be selected as

[TABLE]

Furthermore,

[TABLE]

Let us further investigate a special case $k=t+d$ of (7.7)

[TABLE]

Then, from a derivation similar to (7.3)-(7.6), we have that (4.29) holds for $k=t+d-1$ and

[TABLE]

Therefore,

[TABLE]

Note that the form of (7.9) is different from (7.7) and (7.8). Therefore, we further need deductions.

Assume that for $k\in\{t,...,t+d-2\}$ we have (7.1), (7.2) and

[TABLE]

Similar to (7.3)-(7.6), we have that (4.29) holds for $k$ and $u^{t,x,*}_{k}$ can be selected as

[TABLE]

Furthermore,

[TABLE]

By induction, we can prove (4.29), (4.34) and (4.37).

$(ii)\Rightarrow(i)$. By Lemma 2.3 and reversing the proof of $(i)\Rightarrow(ii)$, we can achieve the result.

B. Proof of Lemma 4.53

By adding to and subtracting

[TABLE]

from $J(t,0;u)$, we have (noting $X^{0}_{t}=0$)

[TABLE]

This completes the proof.

C. Proof of Theorem 5.4

$(i)\Rightarrow(ii)(iii)$. Consider Problem (LQ) for the initial pair $(N-1,\xi)$ with $\xi\in\mathbb{R}^{n}$. Letting $k=N-1$ in (4.12), similar to (7.10) we have

[TABLE]

As (7.11) holds for any $\xi\in\mathbb{R}^{n}$ and any $u_{N-1}\in\mathcal{U}_{ad}^{N-1}$, we must have

[TABLE]

Otherwise, if $W_{N-1}$ has a negative eigenvalue, say $\mu$, then for an eigenvector $\eta$ of $\mu$

[TABLE]

This contradicts the finiteness of Problem (LQ). Further, if $\mbox{Ran}(H_{N-1})\subset\mbox{Ran}(W_{N-1})$, then there exists a $\xi_{0}\in\mathbb{R}^{n}$ such that $H_{N-1}\xi_{0}\in\mbox{Ker}(W_{k})$. Hence,

[TABLE]

This also contradicts the finiteness of Problem (LQ). We therefore have (7.12) and

[TABLE]

Assume that for $k\in\mathbb{T}_{t+d}=\{t+d,...,N-1\}$

[TABLE]

and

[TABLE]

In (7.15), $u^{\ell,\xi,*}$ is given by

[TABLE]

with

[TABLE]

We now prove that (7.14) and (7.15) also hold for the case $\ell=k$. In fact, similarly to (7.10) we have

[TABLE]

which holds for any $\xi\in\mathbb{R}^{n}$ and any $u\in\mathcal{U}_{ad}^{k}$. Let the elements $u_{k+1},...,u_{N-1}$ of $u$ take the following form

[TABLE]

and denote such a $u$ by $\widehat{u}$ with its element $u_{k}$ being freely selected. Then, (7.17) becomes to

[TABLE]

By an analysis similar to that between (7.12) and (7.13), we have

[TABLE]

and

[TABLE]

with $u^{k,\xi,*}$ being given by (5.60). By induction, we can get (ii) and (iii).

$(ii)\Rightarrow(i)$. This is straightforward.

$(ii)\Rightarrow(iii)$. By the proof of Theorem 4.2, we know

[TABLE]

Note that (7.19) holds for any initial pair $(k,\xi)$ and $\ell\in\mathbb{T}_{k}$. We must have

[TABLE]

$W_{k}\geq 0,k\in\mathbb{T}_{t}$, is due to the convexity of $u\mapsto J(t,x;u)$.

$(iii)\Rightarrow(ii)$. This is straightforward from

[TABLE]

where

[TABLE]

$(iii)\Rightarrow(iv)$. Let $(P^{(0)},...P^{(d)})$ be the solution of Riccati-like equation set (4.12)-(4.17) with property $W_{k}W_{k}^{\dagger}H_{k}=H_{k},W_{k}\geq 0,k\in\mathbb{T}_{t}$. From the extended Schur’s lemma, we know $(P^{(0)},...P^{(d)})\in\mathcal{M}$. Hence, $\mathcal{M}$ is nonempty.

$(iv)\Rightarrow(i)$. Let $(P^{(0)},...P^{(d)})\in\mathcal{M}$. Then, from Lemma 5.2, we know

[TABLE]

Hence, Problem (LQ) is finite.