Information based approach to stochastic control problems

TL;DR

This paper introduces an information-based control method for stochastic problems with partial observation, reducing computational complexity by replacing dynamic programming with a sequence of easier control problems.

Contribution

The paper develops an innovative information-based control approach that bounds cost reduction via mutual information and avoids dynamic programming in certain cases.

Findings

IBC can find optimal solutions without dynamic programming

The method reduces computational complexity significantly

It effectively generates system state information during control

Abstract

An information based method for solving stochastic control problems with partial observation has been proposed. First, the information-theoretic lower bounds of the cost function has been analysed. It has been shown, under rather weak assumptions, that reduction of the expected cost with closed-loop control compared to the best open-loop strategy is upper bounded by non-decreasing function of mutual information between control variables and the state trajectory. On the basis of this result, an Information Based Control method has been developed. The main idea of the IBC consists in replacing the original control task by a sequence of control problems that are relatively easy to solve and such that information about the state of the system is actively generated. Two examples of the operation of the IBC are given. It has been shown that the IBC is able to find the optimal solution without using dynamic programming at least in these examples. Hence the computational complexity of the IBC is substantially smaller than complexity of dynamic programming, which is the main advantage of the proposed method.

Equations

$$J_{open}-J_{closed}(\phi )\le \rho (I(X;U|\phi )),$$

$$x_{k+1}=f(x_k,u_k,w_k),k=0,1,...,$$

$$y_k=h(x_k,v_k),$$

$$u_k\in U_{ad},U_{ad}=\{u\in R^r:u_{\min }\le u\le u_{\max }\},$$

$${\phi }_k:R^{m(k+1)}\to U_{ad},u_k={\phi }_k(Y_k),$$

$$\phi :R^{mN}\to U_{ad}^N\subset R^{Nr},U=\phi (Y),$$

$$\phi (Y)=\mathrm{col}({\phi }_0(Y_0),...,{\phi }_{N-1}(Y_{N-1})).$$

$$J(\phi )=E\{L(x_N)|\phi \},$$

$$I(\phi )=H(X|\phi )-H(X|U,\phi ),$$

$$H(X|\phi )=E(-\ln p(X|\phi )),$$

$$H(X|U,\phi )=E(-\ln p(X|U,\phi )).$$

$$\Omega (s)=\{\phi \in S_{ad}:I(\phi )\le s\}$$

$$\underset{\phi \in \Omega (s)}{\inf }J(\phi ).$$

$$J_o=\underset{\phi \in \Omega (0)}{\inf }J(\phi ).$$

$$J_o-J(\phi )\le \rho (I(\phi ))$$

$$\rho (s)=\underset{\phi \in \Omega (s)}{\sup }\left(J_o-J(\phi )\right).$$

$$\rho (0)=\underset{\phi \in \Omega (0)}{\sup }\left(J_o-J(\phi )\right)=J_o-J_o=0.$$

$$J_o-J(\phi )\le \underset{\psi \in \Omega (I(\phi ))}{\sup }\left(J_o-J(\psi )\right)=\rho (I(\phi )),$$

$$d(\Omega (0),\phi )=\underset{\psi \in \Omega (0)}{\inf }||\psi -\phi ||,$$

$$|J(\phi )-J({\phi }_1)|⩽L_J||{\phi }_1-\phi ||,\phi ,{\phi }_1\in S_{ad},$$

$$I(\phi )⩾L_{I}d(\Omega (0),\phi ),\phi \in S_{ad},$$

$$J_o-J(\phi )⩽qI(\phi ),\phi \in S_{ad}.$$

$$\begin{array}{rl}& J(\phi )+qI(\phi )⩾J({\phi }_1)-L_J||{\phi }_1-\phi ||+q{L}_{I}d(\Omega (0),\phi )=\\ & J({\phi }_1)+(q{L}_I-L_J)||{\phi }_1-\phi ||⩾J({\phi }_1)⩾J_o,\phi \notin \Omega (0).\end{array}$$

$$I(\phi )=I(X;U|\phi )⩽I(X;Y|\phi ).$$

$$J(\phi )=E\{-\ln p(x_N|\phi )\}.$$

$$H_o-H(\phi )⩽I(\phi ),\phi \in S_{ad}.$$

$$H_o-H(\phi )⩽\sum _{k=0}^{N-1}I(x_k;u_k|\phi ).$$

$$J(\phi )⩾cn(2\pi e{)}^{-1}{e}^{2n^{-1}(H_o-I(\phi ))}.$$

$$J(\phi )⩾cE|x_1{|}^2⩾c\mathrm{tr}(S)⩾cn|S{|}^{\frac{1}{n}}⩾cn(2\pi e{)}^{-1}{e}^{2n^{-1}H(\phi )}⩾cn(2\pi e{)}^{-1}{e}^{2n^{-1}(H_o-I(\phi ))}.$$

$$x_1=x+u,y=x+v.$$

$$J(\phi )=E\{x_1^2|\phi \}.$$

$${\phi }^{\ast }(y)=-\frac{s_x}{s_x+s_v}y$$

Full text

Keywords: Stochastic control, feedback, information, entropy.

1 Introduction

Optimal controller synthesis in stochastic systems with partial observation can be performed by dynamic programming (DP). Unfortunately, despite the theory of DP is well developed (Zabczyk (1996)), its computational complexity grows exponentially with the number of variables and time steps. As a consequence the problem is practically intractable.

To overcome the curse of dimensionality, a number of approximate methods has been developed. Separation principle and certainty equivalence assumption has been used by Filatov & Unbehauen (2004), Åström & Wittenmark (1995), Tse (1974), BarShalom & Tse (1976). As part of the theory of Partially-Observable Markov Decision Processes (POMDP), various policy-iteration or value-iteration methods were developed by Thrun (2000), Porta et al. (2006), Brechtel et al. (2013), Dolgov (2017), Zhao et al. (2019) and many other researchers. These methods were initially developed for systems with finite number of states and then adopted to more general problems with smooth dynamics. Therefore, as the number of variables and time steps increases, they suffer from the curse of dimensionality. Thus, there is still a need to develop methods with less computational complexity.

Analysis of known optimal solutions (Zabczyk (1996), Filatov & Unbehauen (2004), Åström & Wittenmark (1995), Tse (1974), BarShalom & Tse (1976), Bania (2017)), suggests that active exchange of information between controller and the system is distinctive feature of the optimal controllers (cf. Bania (2018)). Relationships between control of dynamical systems and available information are fundamental for understanding of stochastic control theory. Since the pioneering work of Feldbaum (1965) the connections between control and information theory are intensively studied. Hijab (1984) showed that concept of entropy appears naturally in dual control. The entropic formulation of stochastic control has been given by Saridis (1988) and Tsai et al. (1992). The works of Banek (2010) and Kozlowski & Banek (2011) suggests that information exchange, entropy reduction and stochastic optimality are related to each other. An information and entropy flow in control systems has been analysed in the papers of Mitter & Newton (2005) and Sagawa & Ueda (2013). The controllability, observability and stability of linear control systems with limitations of information contained in the measurements were investigated by Taticonda & Mitter (2004). Touchette & Lloyd (2004) showed that controllability and observability can be defined using the concepts of information theory. One of the most relevant results related to the subject of this article is the inequality of Touchette & Lloyd (2000). They proved that the one-step reduction in entropy of the final state is upper bounded by the mutual information between the control variables and the current state of the system. Delvenne & Sandberg (2013) suggested how to extend this result to more general cost functions.

The main contribution of this paper is as follows. First, the open and closed-loop strategy is defined in terms of mutual information between the system trajectory and control variables. Next, it has been proved under relatively weak assumptions, that

[TABLE]

where, $J_{open}$ is the expectation of the cost corresponding to the best open-loop control, $J_{closed}$ is the expectation of the cost corresponding to any closed-loop strategy $\varphi$ and $I(X;U|\varphi)$ is the mutual information between the system trajectory and control variables under the strategy $\varphi$. Function $\rho$ is non-decreasing and $\rho(0)=0$. Additionally, we prove, that under slightly stronger assumptions, $\rho$ is bounded by linear function. Hence the condition $I(X;U)>0$, is necessary for reduction of the cost below the best open-loop cost. On the basis of inequality (0), an Information Based Control (IBC) has been proposed for finding an approximate solution of stochastic control problems. The phrase ”approximate solution” means that the proposed method is able to find strategy no worse than open-loop feedback optimal (OLFO) algorithm given by Tse (1974). The main idea consists in replacing the original control task by a sequence of control problems that are relatively easy to solve and such that condition $I(X;U)>0$, can be fulfilled. This can be done by introducing a penalty function for information deficiency. As a penalty function, the predicted mutual information between the system trajectory and the measurements has been used. Similar idea has been proposed by Alpcan et al. (2015), however, in this article, the process noise (input disturbances) has been completely ignored, which is very strong and often unrealistic assumption. Additional contributions include sufficient conditions for existing the bounds of type (0), one step information-theoretic bound for quadratic cost and two examples of the operation of IBC. In both examples, the optimal solution has been found analytically by DP and then compared with the IBC solution. It has been shown, that IBC is able to find an optimal solution without using DP, which is the main advantage of the proposed method.

The rest of the paper is organised as follows. Section 2 formulates the stochastic control problem. An information-theoretic lower bounds of the cost are given in section 3. Section 4 presents IBC and section 5 contains an examples of its application. Monte Carlo approximation of the cost function and some computational issues are discussed in section 6. Paper ends with conclusions and list of references.

Notation. Abbreviation $\xi\sim p_{\xi}$ means that variable $\xi$ has a density $p_{\xi}(\xi)$. Symbol $\xi\sim N(m,S)$ means that $\xi$ has normal distribution with mean m and covariance S. If $S>0$ then the density of normally distributed variable is denoted by $N(x,m,S)=(2\pi)^{-\tfrac{n}{2}}|S|^{-\tfrac{1}{2}}\exp(-0.5(x-m)^{T}S^{-1}(x-m))$. Symbol $\mathrm{col}(a_{1},a_{2},...,a_{n})$, denote the column vector. Trace of matrix A is denoted by $\mathrm{tr}(A)$. The inner product of matrices $A$ and $B$ is defined as $\langle A,B\rangle=\mathrm{tr}(A^{T}B)$. Let $\xi\in R^{n}$ and let Q be square matrix of size n. Quadratic form $\xi^{T}Q\xi$ is denoted by $|\xi|_{Q}^{2}$. Entropy of the variable $\xi$ is denoted by $H(\xi)$. Control strategy is denoted by $\varphi$. Symbol $H(\xi|\varphi)$ means that entropy of the variable $\xi$ is calculated with a fixed strategy $\varphi$.

2 Stochastic control task

Let us consider following stochastic system

[TABLE]

where $x_{k}\in R^{n}$, $y_{k}\in R^{m}$, $w_{k}\in R^{n_{w}}$, $v_{k}\in R^{n_{v}}$, $w_{k}\sim p_{w}$, $v_{k}\sim p_{v}$. The inequalities in (3) are elementwise. It’s also possible in some justified cases that $U_{ad}=R^{r}$. Functions $f,h$ are $C^{2}$ wrt. all their arguments. The initial distribution of $x_{0}$ is denoted by $p_{0}^{-}(x_{0})$. Variables $x_{0},w_{0},w_{1},...,w_{k},v_{0},v_{1},...,v_{k}$ are mutually independent for all $k$. Measurements until time $k$ are denoted by $Y_{k}=\mathrm{col}(y_{0},y_{1},...,y_{k})\in R^{m(k+1)}$. Similarly $X_{k}=\mathrm{col}(x_{0},x_{1},...,x_{k})\in R^{n(k+1)}$, $U_{k}=\mathrm{col}(u_{0},u_{1},...,u_{k})\in R^{r(k+1)}$. The control horizon is denoted by $N\geq 1$. We also introduce the following abbreviations $Y=Y_{N-1},U=U_{N-1},X=X_{N-1}$.

Let $B(R^{Nm},R^{Nr})$ be the set of all bounded maps form $R^{Nm}$ into $R^{Nr}$. If $f_{1},f_{2}\in B$, $\alpha,\beta\in R$ then $\alpha f_{1}+\beta f_{2}\in B$. Hence $B$ is linear space. The set $B$ with the norm $\|f\|_{\mathbf{B}}=\sup_{Y\in R^{Nm}}\|f(Y)\|_{R^{Nr}}$, is a Banach space, which will be denoted by $\mathbf{B}$ and call the space of strategy. The measurable map

[TABLE]

is control strategy at time $k$. Let $U_{ad}^{N}=(U_{ad}\times U_{ad}\times,....,\times U_{ad})_{N\mathrm{times}}$. The map

[TABLE]

where

[TABLE]

is admissible control strategy. The set of all admissible strategies is denoted by $S_{ad}$. It follows from (4-6) that $S_{ad}$ is bounded, closed and convex subset of $\mathbf{B}$. Let $L:R^{n}\rightarrow R$ be measurable $C^{2}$ function and let $J:S_{ad}\rightarrow R$ denote the cost functional. We are looking for a strategy $\varphi\in S_{ad}$, that minimizes the functional

[TABLE]

where the expectation is calculated wrt. $x_{0}$, $w_{0},..,w_{N-1}$, $v_{0},...,v_{N-1}$. The optimal strategy will be denoted by $\varphi^{\ast}$ and the abbreviation $J(\varphi^{\ast})=J^{\ast}$ will be used. We will assume that $\varphi^{\ast}$ exists. Optimal control corresponding to realization of $Y_{k}$ will be denoted by $u_{k}^{\ast}=\varphi_{k}^{\ast}(Y_{k})$.

3 Information-theoretic lower bounds of the cost function

If the strategy $\varphi\in S_{ad}$ is fixed, then relations between random variables $X,Y,U$ are described by their joint density $p(X,Y,U|\varphi)$. In particular, if $p(X,U|\varphi)=p(X|\varphi)p(U|\varphi)$, then $X$ and $U$ are independent and information contained in measurements $Y$ is not utilized. This is open-loop control strategy. Reduction of the cost (7), compared to the open-loop, is possible only if $X$ and $U$ are dependent. The natural measure of dependency is mutual information. We will show below that the cost (7) is lower-bounded by some non-increasing function of mutual information between $X$ and $U$.

3.1 General bounds

The mutual information between $X$ and $U$ is given by

[TABLE]

where the entropies $H(X|\varphi)$, $H(X|U,\varphi)$, are defined in usual way i.e.

[TABLE]

The expected value in (10) is calculated wrt. $X$ and $U$.

Definition 1**.**

The strategy $\varphi$ is an open-loop strategy if, and only if, $I(\varphi)=0$. Otherwise $\varphi$ will be called closed-loop or feedback strategy.

Let $s\in R,s\geq 0$. The set

[TABLE]

contains all strategies for which the information $I(\varphi)$ is not greater than $s$. Let $\varphi\in S_{ad}$ be the constant map. Since $\varphi$ is constant then $U$ and $Y$ are independent and $I(\varphi)=0$. Hence $\Omega(s)$ is non-empty for all $s\geq 0$. Consider now a family of optimization problems

[TABLE]

The optimal solution of (12) will be denoted by $\varphi_{s}^{\ast}$ and it is assumed that $\varphi_{s}^{\ast}$ exist for all $s$. The minimum open-loop cost is defined as

[TABLE]

Lemma 1**.**

If the solution of (12) exists for all $s\geqslant 0$, then there exist non-decreasing, bounded function $\rho:[0,\infty)\rightarrow[0,J_{o}-J^{\ast}],$ $\rho(0)=0,$ such that inequality

[TABLE]

holds for all $\varphi\in S_{ad}$.

Proof.

Let us define

[TABLE]

For every $t,s\geq 0$ we have $\Omega(s)\subset\Omega(s+t)$ hence $\rho$ is non-decreasing. If $s=0$ then by the formula (13) we have

[TABLE]

Since $\varphi\in\Omega(I(\varphi))$, then

[TABLE]

which proves (14). ∎

It follows from (14) that $J(\varphi)<J_{o}\Rightarrow I(\varphi)>0$, but function $\rho$ in (14) can be very irregular. To obtain more accurate bound, additional conditions are needed. Let

[TABLE]

denote the distance between $\Omega(0)$ and $\varphi$.

Theorem 1**.**

If there exists numbers $L_{I},L_{J}>0$, such that

[TABLE]

then there exist number $q>0$, that

[TABLE]

Proof.

Let $\varphi\not\in\Omega(0)$ and let $\varphi_{1}\in\Omega(0)$ be such that $||\varphi_{1}-\varphi||=d(\Omega(0),\varphi)$. If $qL_{I}-L_{J}\geqslant 0$, then on the basis of (18), (19) and (13) we get

[TABLE]

If $\varphi\in\Omega(0)$, then $I(\varphi)=0$ and it follows from (13) that $J(\varphi)\geqslant J_{o}$. Hence (20) holds for all $\varphi\in S_{ad}$. ∎

Remark. Data processing inequality (Cover & Thomas (2006), p.34) says that $I(X;F(Y))\leq I(X;Y)$, for any function $F$. Since $U=\varphi(Y)$ then

[TABLE]

As a consequence, lemma 1 and theorem 1 will still be true if we use $I(X;Y|\varphi)$, instead of $I(\varphi)$.

Since $S_{ad}$ is bounded and closed then Lipschitz continuity assumption (18) is not very restrictive. The assumption (19) says that information must grow linearly with the distance from the set $\Omega(0)$, which seems quite natural and not very restrictive. Let us also note, that $I(\varphi)$ need not to be continuous.

3.2 The entropy reduction of the final state

Let us assume that the cost functional has the form

[TABLE]

We will call $J(\varphi)$ the closed-loop entropy and we will write $H(\varphi)=J(\varphi)$. The minimum open-loop entropy of the final state is denoted by $H_{o}=J(\varphi_{0}^{\ast})$. Touchette & Lloyd Touchette & Lloyd (2000), Touchette & Lloyd (2004), showed that one-step (i.e. $N=1$) entropy reduction compared to the best open-loop strategy is upper bounded by ${I}(x_{0};u_{0}|\varphi)$. Their inequality (in our notation), has the form

[TABLE]

It is fundamental limitation in control systems, but unfortunately, the multi-step ($N>1$) version of (24) is very weak (cf. Touchette (2000), p. 47, equation (3.74)). It only says, that there exist strategy $\varphi$, that

[TABLE]

Since correlations between previous measurements and current control are omitted in (25), it may not be fulfilled for some $\varphi$. However, it is still possible on the basis of (24), to construct some one-step bound for (7).

Theorem 2**.**

Let $J(\varphi)=E\{L(x_{1})|\varphi\}$, $x_{1}\in{R^{n}}$. If $L(x_{1})\geqslant{c}|x_{1}|^{2}$, $c>0$, then

[TABLE]

Proof.

Let $S=\mathrm{cov}(x_{1},x_{1}|\varphi)$. Matrix $S$ fulfils the inequality $\mathrm{tr}(S)\geqslant n|S|^{\tfrac{1}{n}}$, (Cover & Thomas (2006), thm. 17.9.4, p. 680). Since Gaussian distribution maximizes entropy over all distributions with the same covariance, then it can be proved that $|S|\geqslant(2\pi e)^{-n}e^{2H(\varphi)}$, (Cover & Thomas (2006), thm. 8.6.5, p. 254). On the basis of these two inequalities and by using (24) one can obtain

[TABLE]

∎

3.3 Elementary example

To illustrate the problem, let us consider one-dimensional system

[TABLE]

Variables $x$ and $v$ are Gaussian i.e. $x\sim N(0,s_{x}),s_{x}>0$, $v\sim N(0,s_{v}),s_{v}>0$. The cost functional has the form

[TABLE]

The best open-loop strategy is $\varphi_{0}^{\ast}=0$ and $J_{o}=s_{x}$. The optimal strategy is given by linear function of $y$

[TABLE]

and the minimum cost is equal to

[TABLE]

Since $x_{1}$ is Gaussian then its open-loop entropy is given by $H_{o}=\tfrac{1}{2}\ln(2\pi eJ_{o})$ and the inequality (26) yields

[TABLE]

for all $\varphi$. One can check by direct calculation that

[TABLE]

and then $J(\varphi^{\ast})=J_{o}e^{-2I(\varphi^{\ast})}$. Hence the bound (31) is tight. The entropy of $x_{1}$, under optimal strategy, is given by $H(\varphi^{\ast})=\tfrac{1}{2}\ln(2\pi eJ(\varphi^{\ast}))$ and one can check that $H_{o}-H(\varphi^{\ast})=I(\varphi^{\ast})$. Hence, the strategy (29) is also optimal for entropy reduction

4 Information based control

Minimum of $J(\varphi)$ can be found by dynamic programming (DP), but computational complexity of DP grows exponentially with the number of time steps and control variables. As a consequence, DP is often impractical and there is a need to construct approximate methods with lower computational complexity (cf. Filatov & Unbehauen (2004), pp. 14-32, Åström & Wittenmark (1995), pp. 354-370.). It is possible, on the basis of the previous section, to construct such an approximate method. The easiest way to simplify the problem is to replace the original control task with the sequence of open-loop control problems. These control problems consists in minimization of

[TABLE]

where $u^{(k)}=\mathrm{col}(u_{k},...,u_{N-1})$, denote the future control sequence. Minimizer of (33) will be denoted by $\bar{u}^{(k)}(Y_{k})$. To control the system, only the first element of $\bar{u}^{(k)}$ is used and the procedure is repeated in subsequent steps. Hence, the control strategy generated by sequential minimization of (33) has the form

[TABLE]

and this may or may not be a feedback in the sense of definition 1. The above simplification is known as Open Loop Feedback Optimal (OLFO) and it is well known that OLFO does not generate information and can not be optimal, except linear Gaussian systems (cf. section 3.3, example 2 below, Tse (1974), Filatov & Unbehauen (2004)). On the other hand, it follows from section 3 and particularly from (20) and (22), that

[TABLE]

which implies, that every controller better than open-loop, must actively generate information. Generating of information can be enforced by adding to (33), a penalty function for information deficiency. Such penalty function can be constructed by using the mutual information between future states and measurements. It is also possible to use $I(X;U)$ as a penalty, however, calculation of $I(X;U)$ is much more difficult than calculation of $I(X;Y)$. Therefore it is computationally more convenient to use $I(X;Y)$. This is basic idea of the Information Based Control (IBC). Practically realizable implementation of the IBC is as follows. Let $X_{k}^{+}=\mathrm{col}(x_{k+1},...,x_{N-1})$, $Y_{k}^{+}=\mathrm{col}(y_{k+1},...,y_{N-1})$, denote the future states and observations. Let us define for $k=0,1,...,N-2$

[TABLE]

This is the mutual information between $X_{k}^{+}$ and $Y_{k}^{+}$, predicted at time $k$ and conditioned on $Y_{k}$. Since $y_{N}$ is irrelevant from the control point of view then one can assume that $I_{N-1}=0.$ Now, at every time instant we are looking for the minimum of the functional

[TABLE]

where

[TABLE]

The expectation in (37) is calculated wrt. $x_{0}$ and $w_{k},...,w_{N-1}$, but not with reference to $v_{k},...,v_{N-1}$, which substantially simplifies the problem. Minimizer of (37) will be denoted by $\bar{u}^{(k)}$. To control the system only the first element of $\bar{u}^{(k)}$ is used and whole procedure is repeated in subsequent steps. Let us note that $\bar{u}^{(k)}$ depends on $Y_{k}$ as required in (4). As a consequence $X$ depends on $U$ and it’s possible that IBC generates feedback strategy in the sense of definition 1. Minimizer of (37) can be considered as compromise between open-loop control (first term) and learning (second term). The intensity of learning is given by $\nu_{k}$. If $\nu_{k}=0$ then IBC becomes Open-Loop Feedback strategy, which is generally not optimal.

Remark. If the system (1), (2) is linear and the disturbances are additive Gaussian white noises then mutual information in (37) does not depend on control (cf. Bania (2018), theorem 3.1). As a consequence, application of the IBC to linear Gaussian systems with quadratic cost gives well known result i.e. Kalman filter and LQ controller.

5 Examples

5.1 Example 1

To illustrate the main idea of the IBC, let us start from the very simple example of the integrator with unknown gain. Let

[TABLE]

The cost function is given by

[TABLE]

The initial distribution of $\theta$ has the form $P(\theta=-1)=p$, $P(\theta=1)=1-p$, $p\in[0,1]$. Since $\theta$ can be treated as second component of the state vector then (39) can be viewed as a special case of (1) and (2).

The optimal solution, obtained by dynamic programming, has the form

[TABLE]

It follows from (39) and (40) that $x_{2}=0.$ Hence the minimal value of the cost $J^{\ast}=0$. The observation $y_{1}$ contain an information about $\theta$ if, and only if, $u_{0}\neq 0$. Hence $I(y_{1};\theta)>0$ if, and only if, $u_{0}\neq 0$.

The information based solution. Let $\nu_{0}=1$. According to (37), at the first step the following cost

[TABLE]

should be minimized. Calculation of the expectation gives

[TABLE]

We know that $I(y_{1},\theta)>0$ if, and only if, $u_{0}\neq 0$, hence the optimal solution at the first step

[TABLE]

In the second step we minimize

[TABLE]

where

[TABLE]

denote estimate of $\theta$, obtained on the basis of $u_{0}$ and $y_{0}$. Minimization gives

[TABLE]

which is just exactly the optimal solution given by (40). Thus, the IBC method allowed us to find optimal solution, without using dynamic programming.

5.2 Example 2

Due to the various modelling inaccuracies, in real life applications the parameters are not constant, but they are rather a stochastic processes. As an example of the system with parametric noise we will first consider one-dimensional deterministic system

[TABLE]

where $\epsilon(t)$ and $\zeta(t)$ represents changes of the gain and the input disturbances respectively. The control input is denoted by $u(t)\in R$. If we assume that $\epsilon$ is Wiener process and $\zeta$ is white noise, then (41) can be written as a system o two Ito equations

[TABLE]

[TABLE]

Processes $w_{1}(t)$ and $w_{2}(t)$ are mutually independent standard Wiener processes. Parameters $a_{c},b_{c},g_{1c},g_{2c}$, are positive numbers. Observation equation has the form

[TABLE]

where $v_{k}=N(0,s_{v}),s_{v}>0$, $t_{k}=kT_{0}$, $T_{0}>0$. If control is piecewise constant i.e. $u(t)=u_{k},t\in[t_{k},t_{k+1})$, then discrete-time version of (42) and (44) is given by

[TABLE]

where

[TABLE]

[TABLE]

The matrices $A,B,D$ can be calculated by using the well-known discretization rules:

[TABLE]

The input noise is a sequence of mutually independent Gaussian random variables i.e. $w_{k}\sim N(0,I_{2x2})$, where $I_{2x2}$ denote identity matrix of size 2. The initial condition is given by $x_{0}\sim N(m_{0}^{-},S_{0}^{-})$.

The cost functional is given by

[TABLE]

where $x_{k,2}$ denote the second component of $x_{k}$ and $q_{k}\geq 0$, $r_{k}>0$. Since this problem has been solved in Bania (2017), only the main results will be presented and some laborious transformations will be omitted. To simplify the notation, we will skip some of the function’s arguments, in particular instead of $m_{k}(Y_{k},U_{k}),S_{k}(U_{k}),\varphi_{k}(Y_{k})$, we will write briefly $m_{k},S_{k},\varphi_{k}$ etc. It has been shown in Bania (2018), that joint density of $x_{k},Y_{k}$ and the conditional density of $x_{k+1}$ are given by

[TABLE]

where

[TABLE]

Let us note, that equations (55-59) describes the Kalman filter for (45), (46).

5.2.1 The optimal solution

According to (4-6), the strategy $\varphi$ consists of two mappings $u_{0}=\varphi_{0}(y_{0})$ and $u_{1}=\varphi_{1}(y_{0},y_{1})$. The optimal solution can be found by dynamic programming. It has been shown in Bania (2017), that optimal strategy is given by

[TABLE]

[TABLE]

where

[TABLE]

[TABLE]

Matrices $S_{k}$, and vectors $m_{k}$ are given by (56) and (57). The inner product of matrices $A$ and $B$ is denoted by $\langle A,B\rangle=\mathrm{tr}(A^{T}B)$.

5.2.2 The information based solution

We will first calculate the conditional expectation. Let us denote $\xi=q_{1}x_{1,2}^{2}+r_{0}u_{0}^{2}+q_{2}x_{2,2}^{2}+r_{1}u_{1}^{2}$. After calculation of the integrals we get

[TABLE]

where

[TABLE]

The conditional mean $m_{0}$ and covariance $S_{0}$ are given by (56), (57), where $S_{0}^{-}$, $m_{0}^{-}$ are known a priori. Now the mutual information will be calculated. It follows from (53) that

[TABLE]

According to section 4 we have $X_{0}^{+}=x_{1}$, $Y_{0}^{+}=y_{1}$, $Y_{0}=y_{0}$, $u^{(0)}=(u_{0},u_{1})^{T}$. Hence $p(X_{0}^{+},Y_{0}^{+}|Y_{0})=p(x_{1},y_{1}|y_{0})$ and calculation of the integral (36) gives

[TABLE]

By the assumption we have $I_{1}(u^{(1)}|Y_{1})=0$. According to (37), at the first step, we minimize the cost

[TABLE]

After performing calculations we get

[TABLE]

where

[TABLE]

The optimal value of $u_{1}$ is given by minimization of (72) wrt. $u_{1}$

[TABLE]

Substitution of (73) into (72) gives the analogue of equation (62)

[TABLE]

where, for simplicity, the function $J_{0}(u_{0},u_{1}(u_{0},y_{0}),y_{0})$ has been denoted by $\Psi(u_{0},y_{0})$. Minimization of (74) wrt. $u_{0}$ gives $\bar{u}_{0}(y_{0})$, which is the information-based strategy at the first step. After the first step, new information contained in $y_{1}$ is used by the filter (53-59) and the new state and covariance estimates ($m_{1}$ and $S_{1}$) are available. Thus, according to section 4, at the second step we minimize

[TABLE]

where

[TABLE]

and the control value $u_{0}$ (optimal or not) is treated as fixed parameter. After completing the calculations similar as above, we get

[TABLE]

where

[TABLE]

The optimal information-based solution in the second step is given by

[TABLE]

By comparing the formulas (61) and (79), we conclude that $\bar{u}_{1}$ will be equal to the optimal control $\varphi_{1}^{\ast}(y_{0},y_{1})$, provided that $u_{0}$ is equal to the optimal control $\varphi_{0}^{\ast}(y_{0})$. If this last condition is fulfilled then the optimal strategy can be recovered by the IBC. We will show below that it is possible provided that parameter $\nu_{0}$ in (71) is appropriately chosen.

5.2.3 Numerical example

The parameters of the continuous-time system (41-43) were: $a_{c}=1$, $b_{c}=1$, $g_{1c}=g_{2c}=\sqrt{2}$, $s_{v}=0.01$, $T_{0}=0.1$. The parameters of the corresponding discrete-time system (45-51) were equal to: $a_{1}=1.0$, $a_{2}=0.90483$, $a_{3}=b=0.09516$, $d_{1}=0.2$, $d_{2}=9.674\>10^{-3}$, $d_{3}=0.18126$, $d_{4}=6.189\>10^{-4}$. The weights were: $r_{0}=r_{1}=10^{-3}$, $q_{0}=0$, $q_{1}=1$. The initial conditions were equal to $m_{0}=(0,0)^{T}$, $S_{0}=\mathrm{diag}(s_{0,1},s_{0,2})$, $s_{0,1}=5$, $s_{0,2}=0.1$. For simplicity, an assumption was made, that $y_{0}=0$. The results of numerical calculations of functions $R_{0}$, (62) and $\Psi$, (74), are shown in fig. 1.

The optimal control $\bar{u}_{0}$ is ambiguous and is equal to $\pm$2.0352. Although the initial condition is concentrated around zero, optimal control is non-zero. This is dual effect, described first by Feldbaum (1965). Let us observe, that parameter $\nu_{0}$ can be chosen such that function $\Psi$, (74), has minima at the same points as function $R_{0}$, (62). Hence the main conclusion that optimal feedback can be realized by the Information Based Control, at least in this example. It’s important to notice that information based solution has been found without using dynamic programming, which substantially reduces computational complexity.

6 Computational issues and practical implementation of the IBC

Minimization of the cost (37) requires in advance the solution of the following problems:

Calculation of the filtering distribution $p(x_{k}|Y_{k})$. 2. 2.

Calculation of the expectation in (37). 3. 3.

Calculation of the mutual information (36).

Filtering distribution can be calculated by Unscented Kalman Filter (UKF), Particle Filter (PF) or Gaussian Sum Filter (GSF) (see Särkä (2013), Alspach & Sorenson (1972), for details). Since both the theory and practical implementations of these filters are well developed, we will assume below that $p(x_{k}|X_{k})$ or its approximation is known. Let $x_{k,i},i=1,...,n_{s}$ denote samples from $p(x_{k}|Y_{k})$ and let $x_{N,i}$, $Y_{k,i}^{+}$ denote the final state and observations generated by (1), (2) with initial condition $x_{k,i}$. Then it is easy to observe that samples $x_{N,i}$, $Y_{k,i}^{+}$ are drawn from $p(x_{N}|Y_{k},u^{(k)})$ and $p(Y_{k}^{+}|Y_{k},u^{(k)})$ respectively. Hence, the Monte Carlo approximation of the expectation in (37) is given by

[TABLE]

Calculation of the mutual information (36) cannot be easily done without additional simplifications. Therefore, below we will briefly discuss special cases that are relatively easy to solve. Let us assume that equation (2) has the form

[TABLE]

where $v_{k}\sim N(0,S_{v})$. By direct calculation we get

[TABLE]

where $n_{k}$ denote size of $Y_{k}^{+}$ and

[TABLE]

is an entropy of $Y_{k}^{+}$, predicted at time $k$. Kernel Density Estimator (KDE) of $p(Y_{k}^{+}|Y_{k},u^{(k)})$ has the form

[TABLE]

where $I_{n_{k}}$ is an identity matrix of size $n_{k}$ and the bandwidth parameter is given by

[TABLE]

Now, the entropy estimator can be constructed as follows

[TABLE]

where

[TABLE]

Combining (86) and (82) we get

[TABLE]

On the basis of (37), (80) and (88)

[TABLE]

Since the last term in (89) does not depend on $u^{(k)}$ then finally, the cost function to be minimized is given by

[TABLE]

Convergence conditions for (84) and (86) are given in Jiang (2017) and Joe (1989). These conditions can be fulfilled assuming that $p_{w}$, $p_{v}$, $f$, $h$ are sufficiently regular. In particular, if $p(Y_{k}^{+}|Y_{k},u^{(k)})$ is bounded, globally Lipschitz, $C^{4}$ and its second order partial derivatives are all upper bounded by integrable function, then (84) converges uniformly and the variance of (86) tends to zero as $n_{s}\rightarrow\infty$. The convergence rate is $O(n^{-\alpha})$, $\alpha\in(0,\tfrac{1}{2}]$.

Since $f,h,L$ are $C^{2}$ then cost (90) is also $C^{2}$ wrt. $u^{(k)}$ and its gradient can be effectively calculated by solving associated adjoint equation. Then minimization of (90) can be performed by combining global search algorithms (e.g. Differential Evolution, Simulated Annealing, Genetic Algorithms) with stochastic quasi-newton methods as local solvers (Byrd et al. (2016)).

Control of linear system with finite number of unknown parameters and with quadratic cost function is another special case that is tractable by the IBC. An analytical formulas describing the cost function and the filtering distribution has been given in Bania (2018). Various types of recursive filters are also analysed in Bania & Baranowski (2016), Bania & Baranowski (2017), Baranowski et al. (2017). Computationally effective lower bound of the mutual information (36), that can be utilized to construct an upper bound of the cost, is given in Bania (2019). Thus, in this particular case, the cost (37) and its gradient can be calculated without using Monte Carlo sampling and the control problem is relatively easy to solve.

7 Conclusions

Lower bounds of the cost function in stochastic optimal control problems have been analysed in terms of information exchange between the system and the controller. It has been proved, under weak assumptions, that the cost function is lower bounded by some decreasing function of mutual information between the system trajectory and control variables. Under some additional regularity conditions, the lower bound obtained above is linear function of information, but the constant $q$ appearing in (20) depend on system dynamics. It also follows from theorem 1 and (22), that minimum value of the cost is determined by the capacity of the measurement channel (i.e maximal value of $I(X;Y)$). Next, on the basis of Touchette-Lloyd inequality, a new one-step lower bound (26) has been established, provided that cost function is quadratic. This bound is independent on system dynamics and in that sense universal.

Inequalities (20) and (22) indicates that restrictions in communication between parts of the system prevent certain states from being reached. One of the examples of such phenomenon is synchronization in dynamical networks. Since the synchronization problem can be interpreted as stochastic control task, then communication constraints of the form $I(X;Y)<C$ implies that $J_{o}-J(\varphi)\leqslant C$. As a consequence synchronization may be lost if $C$ is too small. This was confirmed in Huang et al. (2012).

The conclusion resulting from the analysis of information-theoretic bounds is that feedback controller must actively (if possible) generate information about the state of the system. On the basis of these results, the Information Based Control approach to stochastic control has been proposed. The main idea of the IBC consists in replacing of the original control problem by sequence of simpler, auxiliary control problems. The cost function to be minimized in these auxiliary problems consists in two parts: the predicted expectation of the cost conditioned on available measurements and the penalty function for information deficiency. As penalty function, the predicted mutual information between the trajectory and measurements has been used. Hence the method enforces active generation of information about the system state and is able to generate feedback strategy. The IBC method can be also viewed as modification of the OLFO (Tse (1974)) algorithm or as compromise between control and state estimation.

It follows from section 6 that minimization of the cost (37) can be performed by standard optimization algorithms, without using dynamic programming. Hence the computational complexity of the IBC is substantially smaller than complexity of DP. This feature of the IBC makes the possibility of solving large-scale tasks, which is impossible with DP. It has been shown that IBC is able to find an optimal solutions, provided that learning intensity (parameter $\nu_{k}$) is appropriately selected. The optimal value of $\nu_{k}$ can be tuned experimentally but, at the current stage of research, this problem is not resolved. The ability of the IBC to find optimal solution is surprising but, due to the complexity of the problem, convergence to optimal solution is difficult to investigate and has not been proven.

Effective calculation of the mutual information or development of its approximation is crucial issue and some methods from the optimal experimental design and fault detection theory can be adopted here (see Bania (2019), Uciński (2004), Korbicz et al. (2004)). It is also possible to use the information lower bound proposed by Kolchinsky & Tracey (2017).

Application of the IBC method to solve more realistic control problems and developing of information-based model predictive control algorithms is planned as a part of future works.