The Adaptive LQ Regulator

TL;DR

This paper investigates the optimal adaptive control of linear systems with infinite horizon LQ costs, identifying conditions for certainty equivalence and adaptive stabilizability.

Contribution

It characterizes when certainty equivalence holds and when linear systems are adaptively stabilizable, advancing understanding of adaptive control in partially observed systems.

Findings

Certainty equivalence property holds for specific partially observed systems.

A linear system is adaptively stabilizable iff it is uniformly stabilizable.

Identifies classes where the certainty equivalence value function satisfies Bellman equation.

Abstract

The optimal adaptive control of a linear system in a signal-plus-noise setting with infinite horizon LQ regulator cost is studied. The class of partially observed linear systems for which the certainty equivalence property holds is identified. It is also shown that a linear system is adaptively stabilizable if and only if it is uniformly stabilizable, and the class of partially observed linear systems for which the certainty equivalence value function is a supersolution of the Bellman equation is identified.

Equations

$$p_j(t)=P(\theta =j\mid y(s),0\le s\le t)$$

$$(x_1(t),\dots ,x_N(t),p_1(t),\dots ,p_N(t)),$$

$$H(p)=-\sum _{j=1}^N{p}_j\log p_j\Rightarrow -\frac{{\partial }^{2}H}{\partial p_j\partial p_k}=\frac{{\delta }_{jk}}{p_j}.$$

$$\dot{x}(t)=A_{\theta }x(t)+B_{\theta }u(t),x(0)=x_{\theta },$$

$$y(t)={\int }_0^t{G}_{\theta }x(s)ds+w(t),$$

$$J(x,p,u)=\frac{1}{2}{E}^P\left({\int }_0^{\infty }\left(|C_{\theta }x{|}^2+|u(t){|}^2\right)dt\right).$$

$$0=C^{\ast }C+A^{\ast }K+KA-KB{B}^{\ast }K$$

$$U_{ce}(x,p)=\frac{1}{2}\sum _{j=1}^N{p}_j⟨K_j{x}_j,x_j⟩=\frac{1}{2}{E}^P(⟨K_{\theta }{x}_{\theta },x_{\theta }⟩),$$

$$u(t)=-B_{\theta }^{\ast }{K}_{\theta }x(t).$$

$$\dot{x}(t)=\theta u(t),x(0)=x_{\theta }\in \mathbf{R},$$

$$\frac{1}{2}{\int }_0^{\infty }(p{x}_{+}(t{)}^2+(1-p)x_{-}(t{)}^2+u(t{)}^2)dt.$$

$$0=\theta x(\infty )=\theta x_{\theta }+{\int }_0^{\infty }u(t)dt,$$

$$U(x_{+},x_{-},p)=\{\begin{array}{ll}\frac{1}{2}(p{x}_{+}^2+(1-p)x_{-}^2),& p=0,1\text{ or }{x}_{+}=-x_{-},\\ \infty ,& p\ne 0,1\text{ and }{x}_{+}\ne -x_{-}.\end{array}.$$

$$F{G}_j=B_j^{\ast }{K}_j,j=1,\dots ,N,$$

$$\mathcal{L}f=\frac{1}{2}\sum _{j,k=1}^N\frac{{\partial }^{2}f}{\partial p_j\partial p_k}{p}_j{p}_k⟨z_j-\hat{z},z_k-\hat{z}⟩,$$

$$\begin{array}{rl}\mathcal{H}(x,p,D)& =\underset{u\in {\mathbf{R}}^i}{\min }\left(\frac{1}{2}|u{|}^2+\sum _{j=1}^N\frac{1}{2}{p}_j|C_j{x}_j{|}^2+⟨D_j,A_j{x}_j+B_{j}u⟩\right)\\ & =\frac{1}{2}\sum _{j=1}^N{p}_j|C_j{x}_j{|}^2+\sum _{j=1}^N⟨D_j,A_j{x}_j⟩-\frac{1}{2}{\left|\sum _{j=1}^N{B}_j^{\ast }{D}_j\right|}^2.\end{array}$$

$$-\mathcal{L}f=\mathcal{H}(x,p,\nabla f),$$

$$V_{ce}(x,p)=\frac{1}{2}\sum _{j=1}^N{p}_j⟨K_j{x}_j,x_j⟩+H(p),$$

$${\int }_0^t|u(s){|}^{2}ds<\infty ,t\ge 0,$$

$$w(t)=y(t)-{\int }_0^{t}z(s)ds,t\ge 0,$$

$$J(x,p,u)=\sum _{j=1}^N{p}_{j}J(x,{\delta }_j,u),p\in S^N.$$

$$J(x,p,u)\ge U_{ce}(x,p).$$

$$U(x,p)=\underset{u}{\inf }J(x,p,u).$$

$$\sum _{j=1}^N{p}_j\left|\frac{\partial U}{\partial p_j}(x,p)\right|\le U(x,p)$$

$$\hat{z}(t)=\sum _{j=1}^N{p}_j(t)z_j(t),$$

$$u(t)=-F\hat{z}(t),t\ge 0.$$

$$U(x,p)=\underset{u}{\inf }\left(\sum _{j=1}^N{p}_j{J}_j(x_j,u)\right).$$

$$\sum _{j=1}^N\left|{\nabla }_j\sqrt{U(x,p)}\right|\le \underset{\begin{array}{c}|\xi |\le 1\\ 1\le j\le N\end{array}}{\max }\sqrt{U_j^F(\xi )}$$

$$V(x,p)=\underset{u}{\inf }\left(J(x,p,u)+E^P(H(p(\infty )))\right)$$

$$U(x,p)\le V(x,p)\le U(x,p)+H(p).$$

Full text

{amscode}

93E20, 60G35, 93E35, 34H15, 49L25

1 Introduction

Adaptive control [1, 5, 13, 15, 20, 21, 25, 27, 30] is the control of systems in the presence of system parameter uncertainty, which we assume is the result of observations $y$ of a signal $z$ corrupted by noise $w$, in a signal-plus-noise setting.

Adaptive control problems may be formulated in a non-Bayesian context, with no prior distribution on the unknown system parameter $\theta$ or the noise $w$, or in a Bayesian context, where there is a prior distribution.

In the former case, one designs control laws using a maximum likelihood estimate $\hat{\theta}(t)$ given past observations $y(s)$, $0\leq s\leq t$. In the latter case, one designs control laws using the conditional probabilities

[TABLE]

that the system is in regime $j$, based on past observations. These are computed via Bayes’ rule, which then expresses the conditional expectation $\hat{\theta}(t)$ as an average likelihood estimate.

A certainty equivalent control law is one designed in two stages: resolving regime uncertainty followed by the application of a control law corresponding to a known regime. In the former case, this means implementing the optimal control law corresponding to the parameter estimate $\hat{\theta}(t)$, and, in the latter case, implementing the optimal control law corresponding to a given parameter, averaged over all parameter values. The certainty equivalence property is the fortuitous occurence when the certainty equivalent control law turns out to be optimal, or stable, or both, for the uncertain system.

The adaptive LQ regulator, a generalization of the LQ regulator [2, 4, 6, 7, 17, 28], is an optimal control problem with linear dynamics and quadratic cost, where the system matrices, collectively denoted $\theta$, are not known to the controller, and thus the state $x(t)$ is partially observed.

Any partially observed optimal control problem is equivalent [9, 11, 22, 26] to a completely observed optimal control problem with a much larger state space. For the adaptive LQ regulator, the equivalent completely observed problem has nonlinear dynamics and non-quadratic cost.

We assume there are only a finite number $N$ of operating regimes. Then the equivalent completely observed problem has state

[TABLE]

where $x_{j}(t)$ is the system state in regime $j$, $p_{j}(t)$ is as above, and optimal controls are determined by dynamic programming, which in the current setting reduces to the Bellman differential equation.

In this paper, we isolate a class of linear systems for which the adaptive LQ regulator is completely and explicitly solvable, and where the certainty equivalence property holds. We also clarify the link between adaptive stabilizability, uniform stabilizability, and supersolutions of the Bellman equation.

By introducing an entropy factor into the cost functional, we identify the class of linear systems for which the certainty equivalence property holds. Since minimizing entropy is equivalent to maximizing information, introducing this factor is natural. In fact, this introduction of the entropy is a quantitative expression of dual control [3, 10], wherein the controller attempts to directly affect both the system state and the system uncertainty. The fact that the factor then facilitates the explicit solvability of the Bellman equation is the surprise. This approach is then extended to supersolutions of the Bellman equation, leading to stabilizing feedback controls.

That the standard LQ regulator has a linear optimal feedback law is a consequence of the fact that the Bellman equation of the LQ regulator is explicitly solved by the quadratic function $\left\langle{Kx},{x}\right\rangle/2$, where the Kalman gain $K$ is given by the Ricatti equation.

In the same way, ultimately our analysis rests on explicit solutions of the Bellman equation of the adaptive LQ regulator, through the above quadratic function and the simple calculus fact

[TABLE]

Nevertheless, partially observed problems are different enough from completely observed problems to warrant a careful formulation and re-working of the optimal control problem in the partially observed setting.

Another generalization of the LQ regulator is considered in [19]. Here the system dynamics matrices are perturbed by additive noise, resulting in a completely observed optimal control problem whose Bellman equation can be solved by a quadratic function given by an extended Ricatti equation.

In [14], the entropy was introduced to obtain explicit solutions to the Bellman equation, but the class of linear systems for which the certainty equivalence property holds was not identified, and supersolutions were not considered. Moreover the formulation there focused on the sample space of $(\theta,w)$, where $w$ is the observation noise. The approach here, with the focus on the sample space of $(\theta,y)$, simplifies the derivation of the filtering equations and expresses the problem in its natural setting, leading to optimal results.

2 Overview

Let $(A_{j},B_{j},C_{j})$, $j=1,\dots,N$, be minimal (§3) linear systems and let $x_{j}$, $j=1,\dots,N$, be initial states. Let $S^{N}$ denote the set of probabilities $p=(p_{1},\dots,p_{N})$ on $\{1,\dots,N\}$, let $p$ be in $S^{N}$, and let $\theta$ be a random variable valued in $\{1,\dots,N\}$ and distributed according to $p$.

Let $\mathbf{R}^{i}$ and $\mathbf{R}^{o}$ be the euclidean spaces where inputs $u$ and outputs $y$ take their values. We do not specify explicitly the euclidean space where the state $x$ takes its values, and the state space dimension may vary with $j$.

Throughout, $\left\langle{\upsilon},{w}\right\rangle$ denotes the dot product of vectors $\upsilon,w$ in euclidean space, $|\upsilon|^{2}=\left\langle{\upsilon},{\upsilon}\right\rangle$ denotes the length squared, $M^{*}$ is the transpose of $M$, $|M|$ is the norm of $M$, and $\mathop{Re}M=(M+M^{*})/2$.

The system dynamics are

[TABLE]

the observations are given by the signal-plus-noise model

[TABLE]

and the cost starting from $(x,p)=(x_{1},\dots,x_{N},p_{1},\dots,p_{N})$ corresponding to $u$ is

[TABLE]

Here $z(t)=G_{\theta}x(t)\in\mathbf{R}^{o}$ is the signal, the noise $w$ is a Wiener process independent of $\theta$, and $E^{P}$ is the expectation against the underlying statistics $P$. We assume the system is partially observed in the sense that the control $u(t)\in\mathbf{R}^{i}$ depends only on past observations $y(s)$, $0\leq s\leq t$, for each $t\geq 0$, and we seek controls minimizing $J$. This is in contrast with the completely observed case where $u(t)$ is allowed to depend also on $\theta$. This partially observed optimal control problem is the adaptive LQ regulator.

Let $K=K(A,B,C)$ be the unique positive definite solution of the Ricatti equation

[TABLE]

corresponding to the minimal triple $(A,B,C)$. When $\theta\equiv j$, the partially observed problem is the standard LQ regulator. Here the optimal cost starting from $x_{j}$ is $U_{j}(x_{j})=\left\langle{K_{j}x_{j}},{x_{j}}\right\rangle/2$ where $K_{j}=K(A_{j},B_{j},C_{j})$, $\bar{A}_{j}=A_{j}-B_{j}B^{*}_{j}K_{j}$ is stable, and the optimal control is given by the feedback $u(t)=-B^{*}_{j}K_{j}x(t)$. It follows that, if we allow controls in (2.1), (2.2), (2.3) to depend also on $\theta$, the optimal cost is

[TABLE]

and the optimal control satisfies

[TABLE]

For the partially observed problem, things can go wrong even in the simplest cases. For example, suppose $\theta=\pm 1$ and consider the scalar integrator

[TABLE]

with zero signal, $G_{\theta}\equiv 0$. Then we may restrict to deterministic controls (see (2.16) below) and the cost $J(x_{+},x_{-},p,u)$ reduces to

[TABLE]

Here $x_{+}=x_{+}(0),x_{-}=x_{-}(0)$ are the two possible initial states and $p=P(\theta=1)$. Let $U(x_{+},x_{-},p)=\inf_{u}J(x_{+},x_{-},p,u)$ be the value function. The optimal feedback when $p(1-p)=0$ is $u(t)=-\theta x(t)=-e^{-t}\theta x_{\theta}$. This control is deterministic iff $\theta x_{\theta}$ is deterministic. More generally, if $u$ is any deterministic control with finite cost, $J(x_{+},x_{-},p,u)<\infty$, then, as we will see, $u$ is stabilizing, $x(\infty)=0$, hence

[TABLE]

which implies $\theta x_{\theta}$ is deterministic. Thus, if $x_{+}\not=-x_{-}$ and $J(x_{+},x_{-},p,u)<\infty$, then $p(1-p)=0$. If $x_{+}=-x_{-}$, then $\theta x_{\theta}$ is deterministic and $u(t)=-\theta x(t)$ is deterministic. Since this $u$ is the optimal LQ regulator feedback, we conclude

[TABLE]

This system is strongly not stabilizable.

In the positive direction, we have the following results: The linear system ensembles $(A_{j},B_{j},C_{j},G_{j})$, $j=1,\dots,N$, for which the certainty equivalence property holds for the adaptive LQ regulator are those satisfying

[TABLE]

for some linear feedback $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$.

Given $f=f(x_{1},\dots,x_{N},p_{1},\dots,p_{N})$, let

[TABLE]

where $z_{j}=G_{j}x_{j}$, and $\hat{z}=\sum_{j=1}^{N}p_{j}z_{j}$. Let $D_{j}=\nabla_{j}f$ denote the gradient with respect to $x_{j}$, $D=(D_{1},\dots,D_{N})$, $\nabla f=(\nabla_{1}f,\dots,\nabla_{N}f)$, and let

[TABLE]

Then the Bellman equation of the adaptive LQ regulator is

[TABLE]

in the sense that the optimal cost corresponding to (2.1), (2.2), (2.3) satisfies (2.8).

Let

[TABLE]

where $H$ is given in (1.3). Then the validity of (2.6) for some partial isometry $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ implies $V_{ce}$ is a solution of (2.8). Moreover, under a mild restriction, the converse holds. These last facts are a purely algebraic computation using (2.4), independent of the probabilistic set-up (2.1), (2.2), (2.3). The rest of this section consists of the details surrounding these assertions.

Let $(x,p)$ be an initial state ensemble $x=(x_{1},\dots,x_{N})$ and probability $p=(p_{1},\dots,p_{N})$. A control $u$ is admissible at $(x,p)$ (§6) if $u(t)$ depends only on past observations $y(s)$, $0\leq s\leq t$, for each $t\geq 0$, and there exists statistics $P$ on the sample space of $(\theta,y)$ under which (2.1), (2.2) hold, in the sense that under $P$,

[TABLE]

almost surely, $\theta$ is distributed according to $p$, and

[TABLE]

is a Wiener process independent of $\theta$.

There is no reason for a sufficiently general control $u$ to be admissible: the statistics $P$ may not exist. In §4, we show for each control $u$ and initial condition $(x,p)$, there is at most one $P$. Subsequently, in §6, §7, we show that for each $(x,p)$ and for the specific feedback controls we seek, $P$ exists. We also show the class of controls admissible at $(x,p)$, $p$ in the interior of $S^{N}$, does not depend on $(x,p)$ (§6).

Let $P$ be the statistics corresponding to $(x,p)$ and admissible control $u$, and let $\delta_{j}$ be the distribution on $\{1,\dots,N\}$ equal to one at $j$. Then (§4) the conditional probability $P_{j}(A)=P(A\mid\theta=j)$ is the statistics corresponding to $(x,\delta_{j})$ and $u$. If $J(x,\delta_{j},u)$ is the corresponding cost, then $J(x,\delta_{j},u)$ depends only on $j$, $x_{j}$, and $u$, and

[TABLE]

Let $\bar{u}_{j}(t)=E^{P_{j}}(u(t))$ and let $J_{j}$ be the cost corresponding to the $j$-th system. By the Cauchy-Schwarz inequality, $J(x,\delta_{j},u)\geq J_{j}(x_{j},\bar{u}_{j})$, hence

[TABLE]

This is consistent with the fact that $U_{ce}$ is a subsolution of the Bellman equation (2.8).

An admissible control $u$ is stabilizing at $(x,p)$ if $x(t)\to 0$ as $t\to\infty$, almost surely $P$. We use $J$ to find stabilizing admissible controls. For this approach to succeed, it is necessary that $J$ be Lyapunov in the sense that the finiteness of $J(x,p,u)$ implies $u$ is stabilizing at $(x,p)$. In fact (§6) this is so, when $(A_{j},B_{j},C_{j})$, $j=1,\dots,N$, are minimal.

An admissible control $u$ is optimal at $(x,p)$ if $J(x,p,u)=U(x,p)$, where

[TABLE]

If $u$ is optimal at $(x,p)$ and $U(x,p)$ is finite, it follows that $u$ is stabilizing at $(x,p)$. If $U$ is finite at some $(x,p_{0})$ with $p_{0}$ in the interior of $S^{N}$, by (2.11), $U$ is finite on $\{x\}\times S^{N}$, and a standard argument implies111This inequality is not used in this paper.

[TABLE]

almost everywhere on $\{x\}\times S^{N}$.

Given an initial state $x_{j}$ and an admissible control $u$, let $z_{j}(t)=G_{j}x_{j}(t)$ be the signal when $\theta\equiv j$. Let

[TABLE]

where $p_{j}(t)$ is the conditional probability (1.1), $j=1,\dots,N$. A first guess for a stabilizing admissible control is the certainty equivalent feedback control of the form

[TABLE]

If, for each initial condition $(x,p)$, there is a unique stabilizing admissible control $u$ at $(x,p)$ satisfying (2.15), the system is adaptively stabilizable with feedback $F$. If $\bar{A}_{j}=A_{j}-B_{j}FG_{j}$, $j=1,\dots,N$, are stable, the system is uniformly stabilizable with feedback $F$. By specializing $p=\delta_{j}$, $j=1,\dots,N$, we see uniform stabilizability with feedback $F$ is a necessary condition for adaptive stabilizability with feedback $F$. We show (§6) that uniform stabilizability with feedback $F$ implies adaptive stabilizability with feedback $F$, and moreover the admissible $u$ satisfying (2.15) has finite cost $U^{F}(x,p)<\infty$.

Note when (2.6) holds, the feedbacks (2.5), (2.15) are consistent: the latter is the conditional expectation of the former.

When there is no signal, $G_{\theta}\equiv 0$, $\bar{u}_{j}$ does not depend on $j$, so the partially observed problem reduces to seeking a deterministic control $u$ achieving the minimum in

[TABLE]

Let $\nabla_{j}$ denote the gradient with respect to $x_{j}$, and let $U_{j}^{F}(x)$ denote the cost of the $j$-th system corresponding to the feedback $u=-Fx_{j}$, $j=1,\dots,N$. When there is no signal, if $\bar{A}_{j}=A_{j}-B_{j}F$, $j=1,\dots,N$, are stable, differentiating $J(x,p,u)$ and using (2.16) and the Cauchy-Schwarz inequality as in (3.12) implies11footnotemark: 1

[TABLE]

almost everywhere in $x$, for each $p$.

A similar estimate222That such an inequality is plausible is suggested by $f=\sum_{j=1}^{N}p_{j}|x_{j}|^{2}\Rightarrow\sum_{j=1}^{N}\left|\nabla_{j}\sqrt{f}\right|\leq 1$. in general when there is a signal would be very welcome.

The conditional probabilities $p(t)=(p_{1}(t),\dots,p_{N}(t))$ always converge to a limit $p(\infty)$ as $t\to\infty$. One measure of the information contained in the observations is $-E^{P}(H(p(\infty)))$, where $H$ is the entropy (1.3). Then increasing information corresponds to decreasing entropy, which suggests minimizing the modified cost functional $J(x,p,u)+E^{P}(H(p(\infty)))$ instead. This strategy of minimizing a sum of an “operating risk” and a “probing risk” was first discussed for non-Markov models in [10]. As $0\leq E^{P}(H(p(\infty)))\leq H(p)$ by concavity of $H$, this modification has no effect on stability: The cost functional is Lyapunov before the modification if and only if it is Lyapunov after the modification.

Let

[TABLE]

be the modified optimal cost or value function, where the infimum is over admissible controls $u$. Then

[TABLE]

Given $(x,p)$, an admissible control $u$ is optimal at $(x,p)$ if

[TABLE]

If $u$ is optimal at $(x,p)$ and $V(x,p)$ is finite, then $u$ is stabilizing at $(x,p)$.

In §7, we show a nonnegative $C^{2}$ solution of the Bellman equation (2.8) satisfying $f(0,p)=0$ necessarily equals $U$, and a nonnegative $C^{2}$ solution of (2.8) satisfying $f(0,p)=H(p)$ necessarily equals $V$. In general, we conjecture that, when the system is adaptively stabilizable, $f=U$ and $f=V$ are the unique nonnegative viscosity [8] solutions of (2.8) with the prescribed values at $x=0$.

A function $f$ is a supersolution of (2.8) if the right side is no greater than the left side, for all $(x,p)$. If the reverse inequality holds, $f$ is a subsolution. Supersolutions are of interest because they lead to finite cost and hence stabilizing controls (§7). $U_{ce}$ is always a subsolution of the Bellman equation and $U_{ce}$ is never a supersolution of the Bellman equation, unless $N=1$ (§7).

If the system is adaptively stabilizable with feedback $F$, let $U^{F}(x,p)=J(x,p,u)$ and $V^{F}(x,p)=J(x,p,u)+E^{P}(H(p(\infty)))$ where $u$ satisfies (2.15). Let $\mu>0$ be such that $\mathop{Re}\bar{A}_{j}=\mathop{Re}(A_{j}-B_{j}FG_{j})\leq-\mu$ and $1/\mu$ bounds $|B_{j}|$, $|C_{j}|$, $|G_{j}|$, and $|F|$, $j=1,\dots,N$. Then $U^{F}$ and $V^{F}$ are viscosity supersolutions satisfying

[TABLE]

for some constant $c(\mu)$ depending only on $\mu$.

Let $V_{ce}$ be given by (2.9). We say certainty equivalence holds if $V_{ce}$ is a solution of the Bellman equation (2.8). We show (§7) certainty equivalence holds if there is a partial isometry $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ such that (2.6) holds. When this happens, $V=V_{ce}$, the system is adaptively stabilizable with feedback $F$, and for each $(x,p)$, the unique admissible control satisfying (2.15) and stabilizing at $(x,p)$ is the unique admissible control optimal at $(x,p)$.

We say the signal is faithful if for some $1\leq j\leq N$, the linear map $G_{j}$ is surjective. We show, when the signal is faithful and $N\geq 2$, certainty equivalence holds only if there is a partial isometry $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ such that (2.6) holds.

We show $V_{ce}$ is a supersolution of (2.8) if there is a feedback $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ of norm at most one such that (2.6) holds. When this happens, the system is adaptively stabilizable with feedback $F$, and for each $(x,p)$ the unique admissible control $u(t)$ given by (2.15) satisfies

[TABLE]

We show, when $N\geq 2$ and the signal is faithful, $V_{ce}$ is a supersolution only if there is a feedback $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ of norm at most one such that (2.6) holds (§7).

By rescaling $\lambda^{2}V(x/\lambda,p)$, or by replacing $H$ by $\lambda^{2}H$, the same results are valid when the norm of the feedback is at most $\lambda$, hence for any feedback $F$ satisfying (2.6) and any $(x,p)$, (2.15) yields an admissible control stabilizing at $(x,p)$.

In principle, the results of this paper should remain valid if the prior distribution $p=(p_{1},\dots,p_{N})$ of $\theta$ is replaced by a continuous prior distribution $p$ on the space of all linear systems $(A,B,C,G)$.

More generally, the definitions of signal process and statistics and the entropy identity (5.5) in §4, §5 should extend to the general signal-plus-noise setting where $\theta$ is valued in a Polish space.

3 The LQ Regulator

We review the LQ regulator [6, 14] emphasizing aspects that we use in the adaptive setting.

The problem is to determine the control $u(t)\in\mathbf{R}^{i}$ minimizing the cost

[TABLE]

over all (possibly discontinuous) controls $u$, where the state $x(t)$ is given by

[TABLE]

This is the finite or infinite horizon problem according to whether $T<\infty$ or $T=\infty$. In either case, we take as admissible controls any function $u$ on $[0,\infty)$ valued in $\mathbf{R}^{i}$ satisfying (2.10). With standard convergence on $(T,x)$ and weak convergence on $u$, $J$ is continuous in $(T,x)$ and lower-semicontinuous in $(T,x,u)$. It follows that minimizing controls for (3.1) exist, for any starting state $x$ and horizon $T\leq\infty$. Let

[TABLE]

be the optimal cost or value function. Then $U$ is lower-semicontinuous and $U(0,x)=0$. Since $J_{T}(\lambda x,\lambda u)=\lambda^{2}J_{T}(x,u)$, we have

[TABLE]

Moreover $U_{T}(x)$ is increasing as a function of $T$ and nonnegative, and satisfies the dynamic programming property [12]

[TABLE]

for $0\leq t\leq T$.

We assume throughout $(A,B)$ is controllable, for each $x$, there is a control $u$ driving (3.2) to the origin in finite time, and $(A,C)$ is observable, the zero-response output map $x\mapsto Ce^{tA}x$, $t\geq 0$, determines the initial state $x$. In short, we assume the system $(A,B,C)$ is minimal. As is well-known [6, 14], this happens iff $(B,AB,A^{2}B,\dots)$ and $(C/CA/CA^{2}/\dots)$ have full rank.

Let $U(x)\equiv U_{\infty}(x)$ and $J(x,u)\equiv J_{\infty}(x,u)$ be the infinite horizon value function and cost function. Controllability implies the value function $U(x)$ is finite for all $x$, and observability implies $U_{T}(x)>0$ for all $T>0$ and $x\not=0$. It follows that $U(x)$ is proper, $|x|\to\infty$ implies $U(x)\to+\infty$. Then $U(x(t))\to 0$ as $t\to\infty$ implies $x(t)\to 0$ as $t\to\infty$.

By (3.4) with $T=\infty$, for any admissible control $u$,

[TABLE]

for $T\geq 0$. When $J(x,u)<\infty$, this implies $U(x(t))\to 0$ hence $x(t)\to 0$, as $t\to\infty$.

Let $\nabla_{\xi}J$ denote the gradient of $J$ with respect to $x$ in the direction $\xi$, and let $U^{0}_{T}(x)$ denote the cost corresponding to $u=0$. Since $J$ is quadratic in the state trajectory, by the Cauchy-Schwarz inequality,

[TABLE]

which implies

[TABLE]

It follows that

[TABLE]

which implies $U$ is continuous as a function of $(T,x)$. From this and (3.4), $U=U_{T}(x)$ is the unique nonnegative solution of the Bellman equation

[TABLE]

with initial condition $U(0,x)=0$, in the viscosity sense [8].

If $K(t)$, $0\leq t\leq T$, satisfies the Ricatti differential equation

[TABLE]

with initial condition $K(0)=0$, then $U=\left\langle{K(t)x},{x}\right\rangle/2$, $0\leq t\leq T$, satisfies (3.6), hence

[TABLE]

It follows that $K(t)$ is an increasing positive-definite matrix-valued function of $t$. This implies the unique solution $K(t)$ of (3.7) exists and (3.8) holds, for all $t\geq 0$. Combining (3.4), (3.6), and (3.8), we obtain for each horizon $T>0$ the unique optimal control in linear feedback form

[TABLE]

for $0\leq t\leq T$.

Since $U(x)$ is finite everywhere, $K(T)$ increases as $T\to\infty$ to some $K$ satisfying the Ricatti algebraic equation (2.4), hence $\left\langle{Kx},{x}\right\rangle/2$ satisfies

[TABLE]

This implies (Theorem 7.1 with $N=1$) $U=\left\langle{Kx},{x}\right\rangle/2$ and the unique optimal control is

[TABLE]

which implies (2.4) has a unique positive definite solution $K=K(A,B,C)$ corresponding to every minimal triple $(A,B,C)$.

It follows from (3.4) that

[TABLE]

for all admissible controls $u$. Let $K(1)>0$ denote the solution of (3.7) at $t=1$. Then inserting $T=t+1$ into (3.10) and integrating over $t\geq 0$ yields

[TABLE]

for all admissible controls $u$.

Letting $\bar{A}=A-BB^{*}K$ denote the feedback dynamics. Since $J(x,u)=V(x)<\infty$, $x(t)\to 0$ as $t\to\infty$, hence $\bar{A}$ is stable.

Assume the system is stabilizable, i.e. there is a feedback $F$ such that $\bar{A}=A-BF$ is stable. Then the map $u(t)\mapsto u(t)-Fx(t)$, where $x(t)$ satisfies (3.2), is a bijection of the class of all admissible controls. Thus

[TABLE]

where $x(t)$ satisfies

[TABLE]

If $U^{F}(x)$ denotes the cost $J(x,u)$ corresponding to the feedback $u=-Fx$ in (3.2), then repeating the logic leading to (3.5) yields

[TABLE]

valid for any control $u$.

4 Partially Observed Systems

We derive the existence and uniqueness results for the underlying statistics $P$ for the signal-plus-noise model

[TABLE]

Here $w(t)$ is a Wiener process independent of $\theta\in\{1,\dots,N\}$, $z_{j}(t)$, $j=1,\dots,N$, are processes depending only on the past of the observations $y(t)$, and $z(t)=z_{\theta}(t)$.

Let $C([0,\infty),\mathbf{R}^{o})$ be the space of all continuous sample paths $\alpha$ on $[0,\infty)$ valued in $\mathbf{R}^{o}$. Then $C([0,\infty),\mathbf{R}^{o})$ is the sample space for the observed process $y$, and the set $\Omega=\{1,\dots,N\}\times C([0,\infty),\mathbf{R}^{o})$ of sample pairs $\omega=(j,\alpha)$ is the sample space for the model. Under uniform convergence on compact intervals, $\Omega$ is a complete metric space.

Define $y(t):\Omega\to\mathbf{R}^{o}$ and $\theta:\Omega\to\{1,\dots,N\}$ by $y(t,\omega)=\alpha(t)$, $t\geq 0$, and $\theta(\omega)=j$. Let $\mathcal{Y}_{t}=\sigma(y(s),0\leq s\leq t)$, $t\geq 0$, denote the sigma-algebra on $\Omega$ generated by the observations up to time $t$, and let $\mathcal{Y}$ denote the sigma-algebra generated by $\mathcal{Y}_{t}$, $t\geq 0$. Let ${\mathcal{B}}_{t}=\sigma(\theta,\mathcal{Y}_{t})$, $t\geq 0$, and ${\mathcal{B}}=\sigma(\theta,\mathcal{Y})$. Then $\mathcal{Y}_{t}\subset{\mathcal{B}}_{t}$, $t\geq 0$, and ${\mathcal{B}}$ is the Borel sigma-algebra of the metric space $\Omega$. A process is a Borel map on $[0,\infty)\times\Omega$.

In later sections, when we introduce controls, the processes $z(t)$, $w(t)$, and the statistics $P$ (below) will depend on the control $u$ (and the initial condition $(x,p)$). However, the process $y$ and the random variable $\theta$ will always refer to the fixed quantities defined above.

Let $\mathcal{F}_{t}$ be ${\mathcal{B}}_{t}$ or $\mathcal{Y}_{t}$. An $\mathcal{F}_{t}$* process* is an $\mathcal{F}_{t}$ progressively measurable [18] process. Let $P$ be a probability measure on ${\mathcal{B}}$. A process $z(t)$ is continuous under $P$ if the sample paths $z(\cdot,\omega)$ are continuous for $P$ almost all $\omega$ and right-continuous for all $\omega$.

If $z(t)$ is a ${\mathcal{B}}_{t}$ process, there are $\mathcal{Y}_{t}$ processes $z_{j}(t)$, $j=1,\dots,N$, such that $z(t)=z_{\theta}(t)$ on $\Omega$.

A process $z(t)$ is a signal process under $P$ if

[TABLE]

almost surely $P$. A continuous process is a signal process. If $z(t)$ is a signal ${\mathcal{B}}_{t}$ process under $P$, there is a continuous ${\mathcal{B}}_{t}$ process equal almost surely $P$ to $\int_{0}^{t}z(s)\,ds$.

Let $i=\sqrt{-1}$ and let

[TABLE]

$t\geq 0$. A $(P,\mathcal{F}_{t})$ Wiener process is a continuous $\mathcal{F}_{t}$ process $w(t)$ such that $w(0)=0$ and $\ell(t;ie,w)$ is a $(P,\mathcal{F}_{t})$ martingale, for all $e\in\mathbf{R}^{o}$ [18].

Let $p=(p_{1},\dots,p_{N})\in S^{N}$ and let $z(t)$ be a ${\mathcal{B}}_{t}$ process. A probability measure $P$ on ${\mathcal{B}}$ is a statistics corresponding to $p$ and $z(t)$ if under $P$,

1. S1.

$\theta$ is distributed according to $p$, 2. S2.

$z(t)$ is a signal process, and 3. S3.

$y(t)-\int_{0}^{t}z(s)\,ds$ is a ${\mathcal{B}}_{t}$ Wiener process $w(t)$.

Since $\theta$ is ${\mathcal{B}}_{0}$ measurable, (S3) implies $w$ and $\theta$ are independent under $P$.

Let $W$ be Wiener measure on $\mathcal{Y}$. When there is no signal, $G_{\theta}\equiv 0$, we have $P=p\times W$. Let $\delta_{j}$ be the distribution on $\{1,\dots,N\}$ equal to one at $j$, $j=1,\dots,N$.

Theorem 4.1**.**

Let $\tau$ be a ${\mathcal{B}}_{t}$ stopping time. Then for any statistics $P$ corresponding to $p$ and $z(t)$, there is a statistics $P_{\tau}$ corresponding to $p$ and $z(t)1_{t<\tau}$ satisfying $P=P_{\tau}$ on ${\mathcal{B}}_{\tau}$.

Proof.

This is a rephrasing of standard results [29, 6.1.2]. For $\alpha\in C([0,\infty),\mathbf{R}^{o})$ and $t\geq 0$, let $W_{t,\alpha}$ denote Wiener measure pinned at $\alpha$ on $[0,t]$, and let $Q_{t,j,\alpha}=\delta_{j}\times W_{t,\alpha}$. Define

[TABLE]

Then

[TABLE]

for $A\in{\mathcal{B}}$ and $B\in{\mathcal{B}}_{\tau}$. In particular, $P=P_{\tau}$ on ${\mathcal{B}}_{\tau}$ and $P_{\tau}(\theta=j)=p_{j}$, $j=1,\dots,N$. Since $z(t)1_{t<\tau}$ is ${\mathcal{B}}_{\tau}$ measurable, $z(t)1_{t<\tau}$ is a signal process under $P_{\tau}$. Let $w_{\tau}(t)=y(t)-\int_{0}^{t\wedge\tau}z(s)\,ds$. It remains to be shown that $w_{\tau}(t)$ is a $(P_{\tau},{\mathcal{B}}_{t})$ Wiener process.

Since $w(t)=y(t)-\int_{0}^{t}z(s)\,ds$ is a $(P,{\mathcal{B}}_{t})$ Wiener process, $\ell(t;ie,w)$ is a $(P,{\mathcal{B}}_{t})$ martingale, hence $\ell(t\wedge\tau;ie,w_{\tau})=\ell(t\wedge\tau;ie,w)$ is a $(P,{\mathcal{B}}_{t})$ martingale. On the other hand,

[TABLE]

is a $(Q_{\tau,\theta,y},{\mathcal{B}}_{t})$ martingale. By the martingale splicing lemma [29, 1.2.10], $\ell(t;ie,w_{\tau})$ is a $(P_{\tau},{\mathcal{B}}_{t})$ martingale, thus $w_{\tau}(t)$ is a $(P_{\tau},{\mathcal{B}}_{t})$ Wiener process. ∎

Let $P$ be a probability measure on ${\mathcal{B}}$. If $w(t)$ is a $(P,{\mathcal{B}}_{t})$ Wiener process and $z(t)$ is a signal ${\mathcal{B}}_{t}$ process under $P$, let

[TABLE]

$t\geq 0$, where the first integral is an Ito integral (under $P$) [18]. Then $\ell(t;z,w)$ is a continuous ${\mathcal{B}}_{t}$ process.

A process $z(t)$ is local under $P$ if for some $n\geq 1$,

[TABLE]

almost surely $P$. If $z(t)$ is a local ${\mathcal{B}}_{t}$ process, then $I(t;z,w)$, $\ell(t;z,w)$ are $(P,{\mathcal{B}}_{t})$ martingales, $I(\infty;z,w)$, $\ell(\infty;z,w)$ exist, and $I(\infty;z,w)$, $\ell(\infty;z,w)$ have means [math], $1$ respectively.

Let $z(t)$, $z^{\prime}(t)$ be ${\mathcal{B}}_{t}$ processes. If $P$ is a statistics corresponding to $p$ and $z(t)$, and $z^{\prime}(t)-z(t)$ is local under $P$, then, by the Girsanov theorem [18], $dP^{\prime}/dP=\ell(\infty;z^{\prime}-z,w)$ with $w(t)=y(t)-\int_{0}^{t}z(s)\,ds$ defines statistics $P^{\prime}$ corresponding to $p$ and $z^{\prime}(t)$, and $z^{\prime}(t)-z(t)$ is local under $P^{\prime}$.

It follows that $P$ is a statistics corresponding to $p$ and $z(t)$ and $z(t)$ is local under $P$ iff $z(t)$ is local under $Q=p\times W$ and

[TABLE]

Let $r(t)$ be the quantity in (4.2). Every signal process can be localized:

Lemma 4.2**.**

Let $z(t)$ be a ${\mathcal{B}}_{t}$ process. Then there are $\mathcal{Y}_{t}$ stopping times $\tau_{n}$, $n\geq 1$, such that for any $P$ under which $z(t)$ is a signal process, $z(t)1_{t<\tau_{n}}$ is local under $P$, $r(\tau_{n})=n$ on $\tau_{n}<\infty$ almost surely $P$, $n\geq 1$, and $P(\tau_{n}\leq T)\to 0$ as $n\to\infty$ for all $T>0$.

Proof.

Let

[TABLE]

Then $\sigma_{n}^{-}\leq\sigma_{n}^{+}$, $n\geq 1$, are increasing in $n$, and, for each $n\geq 1$, $\sigma_{n}^{\pm}$ are $\mathcal{Y}$ measurable. Moreover, $r(\sigma^{-}_{n})\leq n$ on $\sigma_{n}^{-}<\infty$, and $r(\sigma_{n}^{-})=n$ if in addition $r(\sigma_{n}^{-}+)<\infty$. For $n\geq 1$,

[TABLE]

is a ${\mathcal{B}}_{t}$ process. Let $z_{j,n}(t)$, $j=1,\dots,N$, be the corresponding $\mathcal{Y}_{t}$ processes, and let $r_{n}(t)$ be the quantity in (4.2) corresponding to $z_{n}(t)$. Then $r_{n}(\infty)\leq n$ identically on $\Omega$. Since $r_{n}(t)$ is a $\mathcal{Y}_{t}$ process with continuous sample paths identically on $\Omega$,

[TABLE]

are $\mathcal{Y}_{t}$ stopping times.

If $z(t)$ is a signal process under $P$, then $r(t)<n$ for $t<\sigma_{n}^{-}$, $\sigma_{n}^{-}\to\infty$ as $n\to\infty$, and $\tau_{n}=\sigma_{n}^{-}$ almost surely $P$, hence $P(\tau_{n}\leq T)\to 0$ as $n\to\infty$, for all $T>0$, and $z(t)1_{t<\tau_{n}}$ is local under $P$. ∎

We call $\tau_{n}$, $n\geq 1$, a localizing sequence for $z(t)$. More generally, if $\tau$ is a $\mathcal{Y}_{t}$ stopping time and $P$ is such that $r(t)<\infty$ for $t<\tau$ and $r(\tau)=\infty$ on $\tau<\infty$, almost surely $P$, then the proof shows $z(t)1_{t<\tau_{n}}$ is local under $P$, $r(\tau_{n})=n$ on $\tau_{n}<\infty$ and $\tau_{n}\leq\tau$, both almost surely $P$, and $P(\tau_{n}\leq T<\tau)\to 0$ as $n\to\infty$ for all $T>0$.

That $\tau_{n}$ is a $\mathcal{Y}_{t}$ stopping time, rather than a ${\mathcal{B}}_{t}$ stopping time, is crucial for the derivation of Theorem 4.5 and the filtering equations in the next section. That $\tau_{n}$ depends only on $z(t)$, and not $P$, is crucial for

Theorem 4.3**.**

There is at most one statistics $P$ corresponding to $p$ and $z(t)$.

Proof.

Suppose $P$ and $P^{\prime}$ are statistics corresponding to $p$ and $z(t)$, and let $P_{\tau_{n}}$ and $P^{\prime}_{\tau_{n}}$ be as in Theorem 4.1, corresponding to $p$ and $z(t)1_{t<\tau_{n}}$. Then, by (4.5), $P_{\tau_{n}}=P^{\prime}_{\tau_{n}}$, and $A\in{\mathcal{B}}_{t}$ implies $A_{n}=A\cap\{t<\tau_{n}\}\in{\mathcal{B}}_{\tau_{n}}$, so

[TABLE]

Now send $n\to\infty$. ∎

By similar techniques, one can show the following. Let $\tau$, $\tau^{\prime}$ be stopping times with $P$, $P^{\prime}$ statistics corresponding to $p$ and $z(t)1_{t<\tau}$, $z(t)1_{t<\tau^{\prime}}$ respectively. If $P^{\prime}(\tau\leq\tau^{\prime})=1$, then $P^{\prime}=P$ on ${\mathcal{B}}_{\tau}$. In particular, this implies: If $P_{\tau}$ and $P$ are statistics corresponding to $z(t)1_{t<\tau}$ and $z(t)$ respectively, then $P=P_{\tau}$ on ${\mathcal{B}}_{\tau}$.

Turning to existence, let $p\in S^{N}$ and let $z(t)$ be a ${\mathcal{B}}_{t}$ process. If $z(t)$ is local under $Q=p\times W$, by (4.5) there is a unique statistics $P$ corresponding to $p$ and $z(t)$, with $P$ sharing the same null events with $Q$ in ${\mathcal{B}}$.

Now assume for each $n\geq 1$, $\tau_{n}$ is a stopping time and $P_{n}$ is a statistics corresponding to $p$ and $z(t)1_{t<\tau_{n}}$. Assume further the sequence is almost surely increasing in the sense

[TABLE]

Then $P_{n}$, $n\geq 1$, are consistently defined on ${\mathcal{B}}_{\tau_{n}}$, $n\geq 1$.

Theorem 4.4**.**

If

[TABLE]

for all $t\geq 0$, then there is a statistics $P$ corresponding to $p$ and $z(t)$.

Proof.

This is a rephrasing of standard results. Since $P_{n}$ are consistently defined on ${\mathcal{B}}_{\tau_{n}}$, there is a probability measure $P$ on ${\mathcal{B}}$ whose restriction to ${\mathcal{B}}_{\tau_{n}}$ is $P_{n}$ [29, 1.3.5]. It follows that $\theta$ is distributed according to $p$ under $P$, $P(\tau_{n}\leq t)\to 0$ as $n\to\infty$, and $z(t)$ is a signal process under $P$.

Let $e\in\mathbf{R}^{o}$. Since $w_{n}(t)=y(t)-\int_{0}^{t}z_{n}(s)\,ds$ is a $(P_{n},{\mathcal{B}}_{t})$ Wiener process, $\ell(t;ie,w_{n})$ is a $(P_{n},{\mathcal{B}}_{t})$ martingale, hence $\ell(t\wedge\tau_{n};ie,w)=\ell(t\wedge\tau_{n};ie,w_{n})$ is a $(P_{n},{\mathcal{B}}_{t})$ martingale. Since $P=P_{n}$ on ${\mathcal{B}}_{\tau_{n}}$, $\ell(t\wedge\tau_{n};ie,w)$ is a $(P,{\mathcal{B}}_{t})$ martingale. Now send $n\to\infty$. ∎

We use (4.7) in §6 and §7 to establish existence of statistics corresponding to specific feedback controls.

Let $e(t)$ be a signal ${\mathcal{B}}_{t}$ process under a statistics $P$ corresponding to $p$ and $z(t)$. It is natural to define

[TABLE]

$t\geq 0$. Then $I(t;e,y)$ and consequently $\ell(t;e,y)$ are continuous ${\mathcal{B}}_{t}$ processes. By Ito’s Lemma [18], it follows that

[TABLE]

almost surely $P$. When $e(t)$ is a $\mathcal{Y}_{t}$ process, we can do better.

Theorem 4.5**.**

Let $P$ be a statistics corresponding to $p$ and $z(t)$. If $e(t)$ is a signal $\mathcal{Y}_{t}$ process under $P$, there is a continuous $\mathcal{Y}_{t}$ process $I(t;e,y)$ satisfying (4.8) almost surely $P$.

Proof.

Assume first $z(t)$ is local under $P$. By the Girsanov theorem, the probability measure $Q=p\times W$ shares the same null events with $P$ in ${\mathcal{B}}$, and $y(t)$ is a $(Q,{\mathcal{B}}_{t})$ Wiener process. Then $e(t)$ is a signal process under $Q$, hence the $Q$ Ito integral $I(t;e,y)=\int_{0}^{t}\left\langle{e(s)},{dy(s)}\right\rangle$ is a continuous $\mathcal{Y}_{t}$ process satisfying (4.8) on $t\geq 0$, almost surely.

By localizing, for each $n\geq 1$, there is a continuous $\mathcal{Y}_{t}$ process $I_{n}(t;e,y)$ equal to (4.8) on $0\leq t<\tau_{n}$, almost surely $P$. Since for $n<n^{\prime}$, $I_{n}(t;e,y)=I_{n^{\prime}}(t;e,y)$ on $0\leq t<\tau_{n}$, almost surely $P$, by the completeness lemma [29, 4.3.3], there is a continuous $\mathcal{Y}_{t}$ process $I(t;e,y)$ equal to $I_{n}(t;e,y)$ on $0\leq t<\tau_{n}$, hence equal to (4.8) for $t\geq 0$, almost surely $P$. ∎

Lemma 4.6**.**

There is a statistics $P$ corresponding to $p$ and $z(t)$ iff there is a statistics $P_{j}$ corresponding to $\delta_{j}$ and $z_{j}(t)$, for each $j$ satisfying $p_{j}>0$, in which case

[TABLE]

and $P(A\mid\theta=j)=P_{j}(A)$, for $A\in{\mathcal{B}}$.

Proof.

Let $\tau_{n}$, $n\geq 1$, be a localizing sequence for $z(t)$, let $P_{n}$ be the statistics corresponding to $p$ and $z(t)1_{t<\tau_{n}}$, and let $P_{n,j}$ be the statistics corresponding to $\delta_{j}$ and $z_{j}(t)1_{t<\tau_{n}}$, for each $j=1,\dots,N$. Since

[TABLE]

by Theorem 4.4, $P$ exists iff $P_{j}$ exist for all $j$ satisfying $p_{j}>0$. For $A\in{\mathcal{B}}_{T}$ let $A_{n}=A\cap\{T<\tau_{n}\}\in{\mathcal{B}}_{T\wedge\tau_{n}}$. Since

[TABLE]

(4.10) follows. ∎

5 Filtering Equations

Here we derive the filtering equations, without any integrability or moment conditions on the signal $z(t)=z_{\theta}(t)$. For background, see [9, 11, 21, 22, 26].

Let $P$ be statistics corresponding to $p\in S^{N}$ and a ${\mathcal{B}}_{t}$ process $z(t)$. Then $\int_{0}^{t}\left\langle{z_{j}(s)},{dy(s)}\right\rangle$ and the likelihoods

[TABLE]

$j=1,\dots,N$, are continuous $\mathcal{Y}_{t}$ processes, and $\ell(t)\equiv\ell(t;z,y)=\ell_{\theta}(t)$, $t\geq 0$, almost surely.

Let

[TABLE]

$j=1,\dots,N$, and let $\hat{z}(t)$ be given by (2.14). Then $\hat{z}(t)$ is a $\mathcal{Y}_{t}$ process and $p(t)=(p_{1}(t),\dots,p_{N}(t))$ is a continuous $\mathcal{Y}_{t}$ process.

Let $\tau_{n}$, $n\geq 1$, be a localizing sequence for $z(t)$. If $p_{n}(t)$ corresponds to $z(t)1_{t<\tau_{n}}$, then $p_{n}(t\wedge\tau_{n})=p(t\wedge\tau_{n})$, $t\geq 0$, almost surely.

We show $p_{j}(t)$, as defined by (5.1), is the conditional probability that $\theta=j$ given $\mathcal{Y}_{t}$. Then (5.1) is Bayes’ rule.

Theorem 5.1**.**

For $t\geq 0$, $j=1,\dots,N$, and $A\in\mathcal{Y}_{t}$,

[TABLE]

and (1.1) is valid.

Proof.

The validity of (5.2) for all $A\in\mathcal{Y}_{t}$ is the definition of (1.1). For (5.2), let $\Phi$ be a nonnegative $\mathcal{Y}_{t}$ measurable random variable. Since $\theta$ and $y(t)$ are independent under $p\times W$,

[TABLE]

hence

[TABLE]

If $z(t)$ is local under $P$, applying these with $\Phi=1_{A}$ and $\Phi=p_{j}(t)1_{A}$, by (4.5),

[TABLE]

If $z(t)$ is not local under $P$, let $A_{n}=A\cap\{t<\tau_{n}\}$ and localize to get

[TABLE]

then send $n\to\infty$. ∎

Since $p(t)$ is a bounded $(P,\mathcal{Y}_{t})$ martingale, $p(\infty)$ exists almost surely $P$ [18].

Theorem 5.2**.**

Let $\hat{z}(t)$ be given by (2.14). Then

[TABLE]

is a $(P,\mathcal{Y}_{t})$ Wiener process.

Proof.

Since

[TABLE]

almost surely $P$, by Ito’s Lemma [18], $\ell(t;ie,\nu)$ satisfies

[TABLE]

almost surely $P$. Taking the expectation of both sides over $A\in\mathcal{Y}_{s}$, since $w$ is a $(P,{\mathcal{B}}_{t})$ Wiener process, the expectation of the Ito integral vanishes [the integrand is local on finite time intervals]. Since $A\in\mathcal{Y}_{r}$ and $\ell(r;ie,\nu)$ is $\mathcal{Y}_{r}$ measurable, the expectation of the second integral also vanishes. Thus

[TABLE]

for $A\in\mathcal{Y}_{s}$. This shows $\ell(t;ie,\nu)$ is a $(P,\mathcal{Y}_{t})$ martingale for $e\in\mathbf{R}^{o}$, hence $\nu(t)$ is a $(P,\mathcal{Y}_{t})$ Wiener process. ∎

It follows that for any signal $\mathcal{Y}_{t}$ process $e(t)$,

[TABLE]

$t\geq 0$, almost surely $P$.

Theorem 5.3**.**

Under $P$, the conditional probabilities satisfy

[TABLE]

$t\geq 0$, $j=1,\dots,N$, almost surely $P$.

Proof.

Apply Ito’s Lemma to the processes $\ell_{j}(t)$ and $p_{j}(t)$ driven by the $(P,{\mathcal{B}}_{t})$ Wiener process $w(t)$, together with (4.9). ∎

Let $H$ be as in (1.3). The following result, which says the time rate of change of the expected information $-E^{P}(H(p(t)))$, $t\geq 0$, is the mean square conditional variance of the signal $z(t)$, holds as soon as statistics $P$ exist for the signal process $z(t)$, with no moment conditions imposed on the signal.

Theorem 5.4**.**

Let $\tau$ be a $\mathcal{Y}_{t}$ stopping time. Then

[TABLE]

In particular, this is so for $\tau=\infty$.

Proof.

By replacing $\tau$ by $\tau\wedge\tau_{n}$ and sending $n\to\infty$, we may assume $z(t)$ is local. With $\mathcal{L}$ is as in (2.7), (1.3) implies

[TABLE]

where $\hat{z}=\sum_{j=1}^{N}p_{j}z_{j}$. Apply Ito’s Lemma to $H(p(t))$, $0\leq t<\tau$, using (5.4). Then the resulting Ito integral has an integrand that is local under $P$, hence its expectation vanishes. This yields

[TABLE]

∎

6 Stabilizability

A control is a $\mathcal{Y}_{t}$ process $u$ valued in $\mathbf{R}^{i}$.

Let $x=(x_{1},\dots,x_{N})$ and $p=(p_{1},\dots,p_{N})$. If there is a probability measure $P$ on ${\mathcal{B}}$ such that (2.10) holds almost surely $P$, then there is a continuous $\mathcal{Y}_{t}$ process $x_{j}(t)$ satisfying

[TABLE]

$j=1,\dots,N$, almost surely $P$. Then the state and signal processes corresponding to $u$ are $x_{\theta}(t)$ and $z(t)=z_{\theta}(t)=G_{\theta}x_{\theta}(t)$.

A control is admissible at $(x,p)$ if there is a probability measure $P$ on ${\mathcal{B}}$ such that (2.10) holds almost surely $P$, and $P$ is the statistics (§4) corresponding to $p$ and $z(t)$. In this case, we call $P$ the statistics corresponding to $(x,p,u)$. When there is no signal, $G_{\theta}\equiv 0$, every control is admissible at $(x,p)$, and $P=p\times W$.

Let $u$ be a control and let $z(t)$, $z^{\prime}(t)$ be the corresponding signal processes starting from initial state ensembles $x$ and $x^{\prime}$ respectively. Since

[TABLE]

the Girsanov theorem implies there is a statistics corresponding to $p$ and $z(t)1_{t<T}$ iff there is a statistics corresponding to $p$ and $z^{\prime}(t)1_{t<T}$. Since this is true for all $T>0$, by Theorem 4.4, there is a statistics $P$ corresponding to $p$ and $z(t)$ iff there is a statistics $P^{\prime}$ corresponding to $p$ and $z^{\prime}(t)$. Since $P$ and $P^{\prime}$ share the same null events in ${\mathcal{B}}_{T}$, $T\geq 0$, (2.10) holds almost surely $P$ iff it holds almost surely $P^{\prime}$. Thus $u$ is admissible at $(x,p)$ iff $u$ is admissible at $(x^{\prime},p)$. This together with Lemma 4.6 implies a control $u$ is admissible at a single $(x,p)$ with $p$ in the interior of $S^{N}$ iff $u$ is admissible at all $(x,p)$. Thus the class of controls admissible at $(x,p)$, with $p$ in the interior of $S^{N}$, does not depend on $(x,p)$, and (2.11) follows from Lemma 4.6.

Theorem 6.1**.**

Assume $(A_{j},B_{j},C_{j})$, $j=1,\dots,N$, are minimal. Then $J(x,p,u)$ finite implies $u$ is stabilizing at $(x,p)$.

Proof.

(3.10) implies

[TABLE]

almost surely $P$, which implies $x(t)\to 0$ as $t\to\infty$, almost surely $P$. ∎

Theorem 6.2**.**

The system (2.1),(2.2) is uniformly stabilizable with feedback $F$ if and only if it is adaptively stabilizable with feedback $F$. If $U^{F}$ and $V^{F}$ are the costs associated to the feedback (2.15), then (2.18) holds.

Proof.

If the system is adaptively stabilizable with feedback $F$, then by choosing $p=\delta_{j}$, $j=1,\dots,N$, we see it is uniformly stabilizable with feedback $F$. Conversely, suppose $\bar{A}_{j}=A_{j}-B_{j}FG_{j}$, $j=1,\dots,N$, are stable for some feedback $F$, and fix an initial condition $(x,p)$.

Under $Q=p\times W$, $y$ is a Wiener process independent of $\theta$, and the system (2.14), (2.15), (5.3), (5.4), (6.1) of stochastic differential equations driven by $y$ has smooth coefficients, hence there is a unique continuous $\mathcal{Y}_{t}$ process (1.2) starting from $(x,p)$, satisfying (2.14), (2.15), (5.3), (5.4), (6.1), and defined up to a $\mathcal{Y}_{t}$ stopping time $\tau\leq\infty$ with [24]

[TABLE]

almost surely $Q$.

Let $z_{j}(t)=G_{j}x_{j}(t)1_{t<\tau}$, $j=1,\dots,N$, and define $u$ by (2.14), (2.15). Then $u$ and $z_{j}(t)$, $j=1,\dots,N$, are $\mathcal{Y}_{t}$ processes and (2.14), (2.15), (6.1) imply $r(\tau)=\infty$ on $\tau<\infty$ almost surely $Q$, where $r(t)$ is the quantity in (2.10).

The goal is to show there are statistics $P$ corresponding to $p$ and $z(t)=z_{\theta}(t)$, with $\tau=\infty$ almost surely $P$. Because the existence of $P$ depends on $\tau$ being infinite almost surely $P$, and $\tau$ being infinite almost surely presumes the existence of $P$, we seem to be stuck in a circular situation. What saves us is the entropy bound (5.5).

Let $\tau_{n}$, $n\geq 1$, be a localizing sequence for $u$ and let $r(t)$ be the quantity in (2.10). Then $\tau_{n}$ is a $\mathcal{Y}_{t}$ stopping time such that $r(\tau_{n})=n$ on $\tau_{n}<\infty$ and $\tau_{n}\leq\tau$, both almost surely $Q$, and $Q(\tau_{n}\leq T<\tau)\to 0$ as $n\to\infty$, for all $T>0$. Fix $T>0$ and let $u_{n}(t)=u(t)1_{t<T\wedge\tau_{n}}$, $x_{j,n}(t)=x_{j}(t)1_{t<T\wedge\tau_{n}}$, and $z_{j,n}(t)=G_{j}x_{j,n}(t)$, $j=1,\dots,N$. Then $u_{n}$ is a control and $x_{\theta,n}(t)$ equals the state process corresponding to $u_{n}$ starting from $x_{\theta}$, on $0\leq t<T\wedge\tau_{n}$. Since $z_{n}(t)=z_{\theta,n}(t)$ is local under $Q$, there are statistics $P_{n}$ corresponding to $p$ and $z_{n}(t)$ such that $P_{n}$ shares the same null events with $Q$ in ${\mathcal{B}}$. Since $\tau_{n}$, $n\geq 1$, is almost surely increasing (4.6), the statistics $P_{n}$ are consistently defined on ${\mathcal{B}}_{\tau_{n}}$, $n\geq 1$.

Note

[TABLE]

does not depend on $n\geq 1$.

Let $p_{n}(t)=(p_{1,n}(t),\dots,p_{N,n}(t))\in S^{N}$ be the corresponding conditional probabilities. Then

[TABLE]

is a solution of (2.14), (2.15), (5.3), (5.4), (6.1) up to time $T\wedge\tau_{n}$. By uniqueness of solutions of stochastic differential equations, $x_{j}(t\wedge\tau_{n})=x_{j,n}(t\wedge\tau_{n})$, $p_{j}(t\wedge\tau_{n})=p_{j,n}(t\wedge\tau_{n})$, $0\leq t<T\wedge\tau_{n}$, almost surely $Q$, hence almost surely $P_{n}$, for each $j=1,\dots,N$.

By Theorem 5.4,

[TABLE]

With $\bar{A}_{\theta}=A_{\theta}-B_{\theta}FG_{\theta}$, $x(t)=x_{\theta}(t)$ satisfies

[TABLE]

for $t<T\wedge\tau_{n}$, hence

[TABLE]

for $t<T\wedge\tau_{n}$, almost surely $P_{n}$. Since $\bar{A}_{\theta}$ is stable, there is a $\mu>0$ such that $\mathop{Re}\bar{A}_{j}\leq-\mu$ and $1/\mu$ bounds $|B_{j}|$ and $|F|$. Then by Young’s convolution inequality [23], and (6.2),

[TABLE]

for some constant $c_{1}(\mu)$ depending only on $\mu$. By (2.15), this implies

[TABLE]

where now $|G_{j}|\leq 1/\mu$ as well. Since $r(\tau_{n})=n$ on $\tau_{n}<\infty$ almost surely $Q$ hence almost surely $P_{n}$,

[TABLE]

This proves (4.7), hence there are statistics $P$ corresponding to $p$ and $z(t)$ whose restriction to ${\mathcal{B}}_{\tau_{n}}$ is $P_{n}$. By (6.7), $\tau=\infty$ almost surely $P$, hence $u$ is an admissible control. Replacing $P_{n}$ by $P$ in (6.5), (6.6) and sending $n\to\infty$ then $T\to\infty$ implies (2.18), where now $|C_{j}|\leq 1/\mu$ as well, hence $u$ is stabilizing at $(x,p)$. Uniqueness of the admissible control follows from uniqueness of solutions of the system (2.14), (2.15), (5.3), (5.4), (6.1). ∎

7 Explicit Solutions

Given $f=f(x_{1},\dots,x_{N},p_{1},\dots,p_{N})$, let

[TABLE]

where the infimum is over controls admissible at $(x,p)$. When $f(0,p)=0$, $V^{f}=U$, and when $f(0,p)=H(p)$, $V^{f}=V$. We derive a verification theorem suitable for the explicit supersolutions.

Theorem 7.1**.**

Let $f$ be a $C^{2}$ nonnegative supersolution of the Bellman equation (2.8). Then for each initial $(x,p)$, there is a unique admissible control $u$, with statistics $P$, satisfying

[TABLE]

almost surely $P$, with corresponding cost

[TABLE]

If $f$ is a $C^{2}$ nonnegative solution of (2.8), then for each $(x,p)$, $f(x,p)=V^{f}(x,p)$ and the admissible control satisfying (7.2) is optimal at $(x,p)$.

Proof.

Given $u\in\mathbf{R}^{i}$, let

[TABLE]

Let $u$ be any admissible control and let (1.2) be the corresponding processes satisfying (5.4) and (6.1). Ito’s Lemma applied to $f(t)\equiv f(x(t),p(t))$, $t\geq 0$, yields

[TABLE]

for any $\mathcal{Y}_{t}$ stopping time $\sigma$ with $|x(t)|\leq c$, $0\leq t<\sigma$. Let

[TABLE]

Since $f$ is a supersolution of (2.8), adding the last two equations implies

[TABLE]

The rest of the proof is a repetition of that of Theorem 6.2. Under $Q=p\times W$, the system (2.14), (5.3), (5.4), (6.1), (7.2) of stochastic differential equations driven by $y$ has smooth coefficients, hence there is a unique continuous $\mathcal{Y}_{t}$ process (1.2) starting from $(x,p)$, satisfying (2.14), (5.3), (5.4), (6.1), (7.2) and defined up to a $\mathcal{Y}_{t}$ stopping time $\tau\leq\infty$ with [24]

[TABLE]

almost surely $Q$.

Let $z_{j}(t)=G_{j}x_{j}(t)1_{t<\tau}$, $j=1,\dots,N$, and define $u(t)$ by (7.2) on $0\leq t<\tau$ and $u(t)=0$ otherwise. Then $u(t)$ and $z_{j}(t)$, $j=1,\dots,N$, are $\mathcal{Y}_{t}$ processes and (2.14), (2.15), and (6.1) imply $r(\tau)=\infty$ on $\tau<\infty$ almost surely $Q$, where $r(t)$ is the quantity in (2.10).

Let $\tau_{n}$, $n\geq 1$, be a localizing sequence for $u(t)$. Then $\tau_{n}$ is a $\mathcal{Y}_{t}$ stopping time such that $r(\tau_{n})=n$ on $\tau_{n}<\infty$ and $\tau_{n}\leq\tau$, almost surely $Q$, and $Q(\tau_{n}\leq T<\tau)\to 0$ as $n\to\infty$. Using $f\geq 0$ and (7.2), (7.4) with $\sigma=T\wedge\tau_{n}$,

[TABLE]

which implies as before $u$ is an admissible control with corresponding statistics $P$ and satisfying (7.2). Sending $n\to\infty$ then $T\to\infty$ in (7.4) with $\sigma=T\wedge\tau_{n}$ implies $J(x,p,u)\leq f(x,p)<\infty$, hence $x(\infty)=0$, hence (7.3).

For the second statement, if $f$ is a solution of (2.8), then for any admissible control $u$, (7.4) is an equality for all $\sigma$. Inserting $\sigma=\infty$ yields $J(x,p,u)+E^{P}(f(0,p(\infty)))\geq f(x,p)$ for all $u$ which implies $f=V^{f}$, and $u$ given by (7.2) is optimal at $(x,p)$. ∎

Given vectors $\upsilon_{j}$, $j=1,\dots,N$, and $p\in S^{N}$, let $\hat{\upsilon}=\sum_{j=1}^{N}p_{j}\upsilon_{j}$. Then

[TABLE]

Let $\upsilon_{j}=B^{*}_{j}K_{j}x_{j}$, $j=1,\dots,N$. Inserting $f=U_{ce}$ into (2.8) and using (2.4), we see $U_{ce}$ is a subsolution or supersolution of (2.8) if and only if (7.5) is $\geq 0$, for all $x_{1},\dots,x_{N},p_{1},\dots,p_{N}$, or $\leq 0$, for all $x_{1},\dots,x_{N},p_{1},\dots,p_{N}$. It follows that $U_{ce}$ is always a subsolution. If $U_{ce}$ is a supersolution and $N\geq 2$, then setting $x_{k}=0$ for all $k\not=j$ yields $p_{j}(1-p_{j})|B_{j}^{*}K_{j}x_{j}|^{2}=0$, hence $B_{j}^{*}K_{j}=0$. But this contradicts minimality of $(A_{j},B_{j},C_{j})$. Hence $U_{ce}$ is never a supersolution unless $N=1$.

Lemma 7.2**.**

Let $G_{j}$ and $H_{j}$, $j=1,\dots,N$, be linear maps into $\mathbf{R}^{o}$ and $\mathbf{R}^{i}$ respectively. If there exists a linear map $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ of norm at most one such that $H_{j}=FG_{j}$, $j=1,\dots,N$, then

[TABLE]

for all $p\in S^{N}$ and all $x_{j}$, $j=1,\dots,N$. Conversely, if $N\geq 2$ and the linear map $G_{1}$ is surjective, and (7.6) holds, there exists a linear map $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ of norm at most one such that $H_{j}=FG_{j}$, $j=1,\dots,N$. If equality holds in (7.6), then $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ may be chosen a partial isometry.

Proof.

If $H_{j}=FG_{j}$, $j=1,\dots,N$, with the norm of $F$ at most one, then (7.6) is immediate. Conversely, assume (7.6). Choosing $j$ and $x_{k}=0$ for $k\not=j$ implies $|G_{j}x_{j}|^{2}\geq|H_{j}x_{j}|^{2}$ hence there exists $F_{j}$ on $\mathbf{R}^{o}$ such that $H_{j}=F_{j}G_{j}$, $j=1,\dots,N$.

Choosing $j$ and $p_{1}+p_{j}=1$, (7.6) yields $|G_{j}x_{j}-G_{1}x_{1}|^{2}\geq|F_{j}G_{j}x_{j}-F_{1}G_{1}x_{1}|^{2}$. Choosing $x_{1}$ such that $G_{1}x_{1}=G_{j}x_{j}$ implies $F_{1}G_{j}x_{j}=F_{j}G_{j}x_{j}=H_{j}x_{j}$, hence $F=F_{1}$. ∎

Theorem 7.3**.**

$V_{ce}$* is a supersolution of the Bellman equation if there exists $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ of norm at most one satisfying (2.6). Conversely, if $N\geq 2$ and the signal is faithful and $V_{ce}$ is a supersolution, there exists $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ of norm at most one satisfying (2.6). When this happens, the unique admissible control $u$ given by (2.15) satisfies (2.19), and the system is adaptively stabilizable with feedback $F$.*

Proof.

Inserting (2.9) into (2.8) and using (2.4), (5.6), we see $V_{ce}$ is a supersolution of (2.8) if and only if

[TABLE]

holds for all $x_{1},\dots,x_{N},p_{1},\dots,p_{N}$. If there is a feedback $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ of norm at most one such that (2.6) holds, then (7.7) holds. Conversely, if (7.7) holds, then by Lemma 7.2, there is an $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ of norm at most one such that (2.6) holds. The second statement follows from Theorem 7.1. ∎

Theorem 7.4**.**

Certainty equivalence holds if there exists a partial isometry $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ satisfying (2.6). Conversely, if $N\geq 2$ and the signal is faithful and certainty equivalence holds, there exists a partial isometry $F:\mathbf{R}^{o}\to\mathbf{R}^{i}$ satisfying (2.6). When this happens, the cost of the feedback (2.15) starting from $(x,p)$ equals $V_{ce}(x,p)$, and the system is adaptively stabilizable with feedback $F$.

Proof.

Same proof as the previous Theorem except (7.7) is now an equality. ∎

Disclosure statement

No potential conflict of interest was reported by the author.