Optimal Control Problem for Discrete-Time Systems with Colored Multiplicative Noise

TL;DR

This paper addresses the optimal control of discrete-time systems affected by colored multiplicative noise, providing necessary and sufficient conditions for solvability using stochastic difference equations.

Contribution

It introduces a comprehensive analysis of control problems with colored noise and derives solvability conditions for systems with and without input delay.

Findings

Derived necessary and sufficient conditions for control problem solvability.

Analyzed systems with colored multiplicative noise and delay.

Provided a framework for solving stochastic difference equations.

Abstract

The optimal control problem for discrete-time systems with colored multiplicative noise is discussed in this paper. The problem will be more difficult to deal with than the case of white noise due to the correlation of the adjoining state. By solving the forward and backward stochastic difference equations (FBSDEs), the necessary and sufficient conditions for the solvability of the optimal control problems in both delay-free and one-step input delay case are given.

Figures

Equations

$$x_{k+1}=(A_0+A_1{\omega }_k)x_k+(B_0+B_1{\omega }_k)u_{k-d},$$

$$x_{k+1}$$

$$x_{k+1}=A_k{x}_k+B_k{u}_k,$$

$$A_k=A_0+A_1{\omega }_k,$$

$$B_k=B_0+B_1{\omega }_k+B_2{\omega }_{k-1},$$

$$J_N$$

$${\lambda }_{k-1}=E[A_k^{\prime }{\lambda }_k|{\mathcal{F}}_{k-1}]+Q{x}_k,$$

$${\lambda }_N=P_{N+1}{x}_{N+1},$$

$$0=R{u}_k+E[B_k^{\prime }{\lambda }_k|{\mathcal{F}}_{k-1}].$$

$$P_k=A_0^{\prime }{P}_{k+1}{A}_0+{\sigma }^2{A}_1^{\prime }{P}_{k+1}{A}_1+Q-M_k^{\prime }{{\rm Y}}_k^{-1}{M}_k,$$

$${{\rm Y}}_k$$

$$M_k$$

$$P_k$$

$${\bar{M}}_k$$

$$R_k$$

$$u_k=-{{\rm Y}}_k^{-1}{M}_k{x}_k.$$

$$J_N^{\ast }$$

$${\lambda }_{k-1}=P_k{x}_k.$$

$${\bar{J}}_k$$

$${\bar{J}}_N=u_N^{\prime }{{\rm Y}}_N{u}_N>0,$$

$$0$$

$$u_N=-{{\rm Y}}_N^{-1}{M}_N{x}_N,$$

$${\lambda }_{N-1}$$

$$E[x_k^{\prime }{\lambda }_{k-1}-x_{k+1}^{\prime }{\lambda }_k]$$

$$E[x_{m+1}^{\prime }{\lambda }_m-x_{N+1}^{\prime }{P}_{N+1}{x}_{N+1}]$$

$${\bar{J}}_m$$

$$0$$

$$u_m=-{{\rm Y}}_m^{-1}{M}_m{x}_m,$$

$${\lambda }_{m-1}$$

$$V(k,x_k)=E[x_k^{\prime }{P}_k{x}_k],$$

$$V(k)-V(k+1)$$

Full text

I INTRODUCTION

The linear quadratic control was first introduced by Kalman [Kalman, 1960] in 1960, and attracted many other researchers to study it, see [Bismut, 1976], [Chen et al., 1998], [Qi et al., 2017], [Hou et al., 2017], [Ju et al., 2018] and references therein. It is generally known that uncertainty exists universally in practical application so that a renewed problem for systems with stochastic uncertainties has received much attention [Liang et al., 2018], [Rami et al., 2000], [Gershon et al., 2001], [Gao et al., 2017], [Xu et al., 2018] following the pioneering work by Wonham [Wonham, 1968]. For example, [Rami et al., 2000] discussed the indefinite LQ control problem for the discrete time system with state and control dependent noise and established the equivalence between the well-posedness and the attainability of the LQ problem. [Gershon et al., 2001] considered the $H_{\infty}$ optimal control and filtering problem for linear discrete-time systems with stochastic uncertainties for the finite-horizon case.

It is worth noting that the aforementioned works almost considered the system that the coefficient of state and/or control variables involving one multiplicative noise, such as the system in [Zhang et al., 2015] as follows:

[TABLE]

in which $\omega_{k}$ is a white noise with zero mean and variance $\sigma^{2}$. However, when noises $\omega_{k}$, $\omega_{k-1}$ simultaneously involved in the coefficient of control $u_{k-d}$, i.e.,

[TABLE]

$u_{k}$ is the input control with delay $d>0$ and $A,\bar{A},B,\bar{B},\tilde{B}$ are constant matrices with compatible dimensions, it can be seen that the system state is correlated at adjoining times. In order to distinguish the system (2), we call this phenomenon as colored multiplicative noise systems. Actually, this phenomenon exists in many fields such as in engineering field, see [Kay, 1981], [Biswas et al., 1972], [Bryson et al., 1965] and references therein. But due to the correlation of the adjoining state, the LQ problem for the colored noise system with input delay will be more complex to solve.

Recently, some substantial progress for the optimal LQ control has been made by proposing the approach of solving the forward and backward differential/difference equations (FBDEs, for short), see [Zhang et al., 2015], [Zhang et al., 2017] for details. Inspired by these works, we considered the linear quadratic optimal control problem for discrete time colored multiplicative noise system with one-step input delay or without delay. The contributions of this paper are as follows. Firstly, a necessary and sufficient condition for the optimal control problem to admit a unique solution is proposed in terms of the maximum principle. Secondly, the optimal controller and the optimal cost are explicitly presented via a coupled Riccati equation which is derived from the solution to the forward (the state equation) and backward (the costate equation) stochastic difference equations (FBSDEs, for short), which are more difficult to solve due to the correlation of the adjoining state compared with that in [Zhang et al., 2015]. Finally, the non-homogeneous relationship between the costate and the state is established.

The rest of this paper is organized as follows. The solvability of the optimal control problem for the delay-free case is given in Section 2. As to the one-step input delay case, the result is presented in Section 3. Conclusions will be given in Section 4.

Notations: $R^{n}$ stands for the usual $n$-dimensional Euclidean space; $I$ denotes an identity matrix with appropriate dimension; The superscript ′ represents the matrix transpose; Real symmetric matrix $A>0$ (or $\geq 0$) implies that A is strictly positive definite (or positive semi-definite). $\{\Omega,\mathcal{F},P,\{\mathcal{F}_{k}\}_{k\geq 0}\}$ represents a complete probability space, with natural filtration $\{\mathcal{F}_{k}\}_{k\geq 0}$ generated by $\{\omega_{0},\omega_{1},\cdots,\omega_{k}\}$ augmented by all the $P$-null sets. $E[\cdot|\mathcal{F}_{k}]$ means the conditional expectation with respect to $\mathcal{F}_{k}$ and $\mathcal{F}_{-1}$ is understood as $\{\emptyset,\Omega\}$.

II Delay-free Case

Consider the discrete-time stochastic system without delay

[TABLE]

where

[TABLE]

$w_{k}$ is a scalar random white noise with zero mean and variance $\sigma^{2}$, and $A_{0},A_{1},B_{0},B_{1},B_{2}$ are constant matrices with compatible dimensions. And the following cost function:

[TABLE]

where $Q$, $R$ and $P_{N+1}$ are positive semi-definite matrices.

Problem 1:   Find a $\mathcal{F}_{k-1}$ measurable $u_{k}$ such that (4) is minimized subject to (3).

By Pontryagin’s maximum principle, it yields the following costate equations

[TABLE]

with the terminal value

[TABLE]

and the equilibrium condition

[TABLE]

To facilitate the explanation of Problem 1, we will introduce the following difference equation as

[TABLE]

where

[TABLE]

**Remark 1: **   When $B_{2}=0$, the above equation can be reexpressed as

[TABLE]

with

[TABLE]

which is the Riccati equation of white noise case.

The following is the introduction of the main theorem in this section.

Theorem 1:   Problem 1 has a unique solution if and only if $\Upsilon_{k}>0$. In this case, the optimal controller $u_{k}$ is stated as

[TABLE]

The associated optimal value of (3) is given by

[TABLE]

Moreover, the optimal costate $\lambda_{k-1}$ and state $x_{k}$ satisfy the following non-homogeneous relationship

[TABLE]

Proof:  “Necessity”: Assume that Problem 1 admits a unique solution, we will adopt induction to illustrate $\Upsilon_{k}>0$ in (9). For convenience, we show

[TABLE]

Firstly, for $k=N$, from (3), we know that $\bar{J}_{N}$ can be expressed as a quadratic function of $x_{N}$ and $u_{N}$. Letting $x_{N}=0$, then $\bar{J}_{N}=u_{N}^{\prime}\Upsilon_{N}u_{N}$, for the uniqueness of solution to Problem 1, it is clear that the optimal controller is $u_{N}=0$ and the optimal cost is 0. Therefore, for nonzero controller $u_{N}$, we have

[TABLE]

so does $\Upsilon_{N}>0$.

In this case, from (3), (6) and (7), we have

[TABLE]

Therefore, the optimal controller can be computed as

[TABLE]

which is correspond to (14) with $k=N$.

As to $\lambda_{N-1}$, from (3), (6) and (20), we can obtain that

[TABLE]

from (6), it is clear to see $\lambda_{N-1}$ has the same form with (12) with $k=N$.

In order to complete the proof by induction, we take any $m$ with $0\leq m\leq N$. And for all $k\geq m+1$, we make the following assumptions: first, in (9) $\Upsilon_{k}>0$; second, $\lambda_{k-1}=P_{k}x_{k}$ in which $P_{k}$ satisfies (6)-(8); third, $u_{k}=-\Upsilon_{k}^{-1}M_{k}x_{k}$ is the optimal control. In consideration of these assumptions, next we’ll verify that these are all satisfied for $k=m$.

First of all, we will illustrate $\Upsilon_{m}>0$. From (3), (5) and (7), it yields that

[TABLE]

Now we put the above-mentioned formula count up from $k=m+1$ to $k=N$ on both sides, then

[TABLE]

Therefore, we have

[TABLE]

To check $\Upsilon_{m}$, let $x_{m}=0$, thus,

[TABLE]

For any nonzero control $u_{m}$, in view of the uniqueness of optimal control, we can obtain that $\Upsilon_{m}>0$. Next we will show the expression of the optimal $u_{m}$. From (3), (5) and (7), we have

[TABLE]

thus, the optimal control $u_{m}$ can be obtained as

[TABLE]

in which $\Upsilon_{m},M_{m}$ are defined as in (9) and (10).

Next we will investigate the relationship between costate and state in the case of $k=m$. Considering (3), (5) and (27), it yields that

[TABLE]

Seeing that (8)-(10), it implies that the aforementioned equation satisfies (16). By induction, we complete the Necessity proof.

“Sufficiency”: When $\Upsilon_{k}>0$ is satisfied, the unique optimal controller of Problem 1 and the optimal cost functional will be illustrated, respectively.

In order to illustrate the main result more clearly, we first define a function $V(k)$ as

[TABLE]

in which $P_{k}$ is as in (8).

In consideration of (3) and (8)-(10), we can easily calculate that

[TABLE]

Adding from $k=0$ to $k=N$ on both sides of the aforementioned equation, the cost functional (1.2) can be rewritten as

[TABLE]

The condition for the above inequalities to be established is $\Upsilon_{k}>0$. Hence, it’s easy to see that the optimal cost is $J_{N}^{\ast}=E(x_{0}^{\prime}P_{0}x_{0})$. And the optimal control has the same form as in (14). This sufficient proof is completed.

III One-step Delay Case

Considering the following system with one-step input delay:

[TABLE]

The cost functional is

[TABLE]

where $Q$, $R$ and $P_{N+1}$ are positive semi-definite matrices.

Problem 2:   Find a $\mathcal{F}_{k-1}$ measurable $u_{k}$ such that (33) is minimized subject to (31).

By stochastic maximum principle, we can obtain the forward and backward stochastic difference equations with one-step input delay.

[TABLE]

with initial values $x_{0}$ and $u_{-1}$.

Similar to the derivation of the solution to Problem 1, we have the solution to Problem 2.

Theorem 2:   Problem 2 is uniquely solvable if and only if $R_{k}>0$. In this case, the optimal controller $u_{k}$ is stated as

[TABLE]

in which

[TABLE]

with

[TABLE]

with terminal value $P_{N+1}$. Moreover, the optimal costate $\lambda_{k-1}$ and state $x_{k}$ satisfy the following non-homogeneous relationship

[TABLE]

Proof: Following the proof of Theorem 1, the above result can be similarly obtained, so we omit it here.

IV CONCLUSIONS

This paper mainly studied the linear quadratic regulation problem for discrete-time systems with colored multiplicative noise for both delay-free and one-step input delay case. The necessary and sufficient condition for the solvability of optimal control problem was presented by solving the FBSDEs derived from the maximum principle. Moreover, the optimal controller and cost were given in terms of the coupled difference equations developed in this paper. For any input delay $d>0$, the systems are more general but more difficult to deal with. Therefore, the optimal control problem for linear systems with any input delay and colored multiplicative noise are worth considering in the future.