An Optimal Control Derivation of Nonlinear Smoothing Equations

TL;DR

This paper explores the connection between nonlinear smoothing for continuous-time Markov processes and optimal control of the Liouville equation, using a variational approach and relating it to classical estimation methods.

Contribution

It introduces a variational formulation linking nonlinear smoothing with optimal control of the Liouville equation, extending classical estimation techniques to nonlinear settings.

Findings

Derives Hamilton's equations related to nonlinear smoothing.

Connects the optimal control problem to the Zakai equation.

Generalizes Mortensen's minimum energy estimator to nonlinear cases.

Abstract

The purpose of this paper is to review and highlight some connections between the problem of nonlinear smoothing and optimal control of the Liouville equation. The latter has been an active area of recent research interest owing to work in mean-field games and optimal transportation theory. The nonlinear smoothing problem is considered here for continuous-time Markov processes. The observation process is modeled as a nonlinear function of a hidden state with an additive Gaussian measurement noise. A variational formulation is described based upon the relative entropy formula introduced by Newton and Mitter. The resulting optimal control problem is formulated on the space of probability distributions. The Hamilton's equation of the optimal control are related to the Zakai equation of nonlinear smoothing via the log transformation. The overall procedure is shown to generalize the classical Mortensen's minimum energy estimator for the linear Gaussian problem.

Equations

$$Z_t={\int }_0^{t}h(X_s)\mathrm{d}s+W_t$$

$$\mathrm{d}{X}_t=a(X_t)\mathrm{d}t+\sigma (X_t)\mathrm{d}{B}_t,X_0\sim {\nu }_0$$

$$({\cal A}f)(x):=a^{\top}(x)\nabla f(x)+{\frac{1}{2}}\mbox{tr}\big{(}\sigma\sigma^{\top}(x)(D^{2}f)(x)\big{)}.$$

$$({\cal A}^{\dagger}f)(x)=-\operatorname{div}(af)(x)+{\frac{1}{2}}\sum_{i,j=1}^{d}\frac{\partial^{2}}{\partial x_{i}\partial x_{j}}\big{(}[\sigma\sigma^{\top}]_{ij}f\big{)}(x).$$

$$\text{(forward)}:$$

$$p_0(x)$$

$$\text{(backward)}:$$

$$q_T(x)$$

$$\mathsf{P}(X_t\in \mathrm{d}x|{\mathcal{Z}}_T)\propto p_t(x)q_t(x)\mathrm{d}x.$$

$$\mathrm{d}(\log p_t)(x)$$

$$\frac{\partial {\mu }_t}{\partial t}(x)$$

$${\mu }_0(x)$$

$$-\frac{\partial {\lambda }_t}{\partial t}(x)$$

$${\lambda }_T(x)$$

$$\displaystyle\Big{[}\frac{\,\mathrm{d}\mu_{t}}{\,\mathrm{d}t}\Big{]}_{i}$$

$$\displaystyle-\Big{[}\frac{\,\mathrm{d}\lambda_{t}}{\,\mathrm{d}t}\Big{]}_{i}$$

$$\min_{\tilde{P}}\quad{\sf E}_{\tilde{P}}\Big{(}\log\frac{\,\mathrm{d}\tilde{P}}{\,\mathrm{d}P}\Big{)}-{\sf E}_{\tilde{P}}\Big{(}\log\frac{\,\mathrm{d}Q^{z}}{\,\mathrm{d}P}\Big{)}.$$

$$\min_{\tilde{P}}\quad{\sf D}(\tilde{P}\|P)+{\sf E}\Big{(}\int_{0}^{T}z_{t}\,\mathrm{d}h(\tilde{X}_{t})+{\frac{1}{2}}|h(\tilde{X}_{t})|^{2}\,\mathrm{d}t-z_{T}h(\tilde{X}_{T})\Big{)}.$$

$$\,\mathrm{d}\tilde{X}_{t}=a(\tilde{X}_{t})\,\mathrm{d}t+\sigma(\tilde{X}_{t})\big{(}u_{t}(\tilde{X}_{t})\,\mathrm{d}t+\,\mathrm{d}\tilde{B}_{t}\big{)},\quad\tilde{X}_{0}\sim\pi_{0}$$

$$(\tilde{\mathcal{A}}(v)f)(x)$$

$$({\tilde{\mathcal{A}}}^{†}(v)f)(x)$$

$$\mathcal{L}(\rho ,v;y):=\frac{1}{2}⟨\rho ,|v{|}^2+h^2⟩+y⟨\rho ,\tilde{\mathcal{A}}(v)h⟩.$$

$${\mathrm{Min}}_{{\pi }_0,u}:\mathsf{J}({\pi }_0,u;z)=\mathsf{D}({\pi }_0\Vert {\nu }_0)-z_T⟨{\pi }_T,h⟩+{\int }_0^T\mathcal{L}({\pi }_t,u_t;z_t)\mathrm{d}t$$

$$\text{Subj.}:\frac{\partial {\pi }_t}{\partial t}(x)=({\tilde{\mathcal{A}}}^{†}(u_t){\pi }_t)(x).$$

$$\frac{\partial {\pi }_t}{\partial t}(x)$$

$$-\frac{\partial {\lambda }_t}{\partial t}(x)$$

$${\lambda }_T(x)$$

$${\pi }_0(x)=\frac{1}{C}{\nu }_0(x)e^{{\lambda }_0(x)}$$

$$u_t(x)={\sigma }^{\top }(x)\nabla ({\lambda }_t-z_{t}h)(x).$$

$$[\tilde{A}(v){]}_{ij}=\{\begin{array}{ll}[A{]}_{ij}[v{]}_{ij}& i\ne j\\ -{\sum }_{j\ne i}[\tilde{A}(v){]}_{ij}& i=j\end{array}$$

$$[C(v){]}_i=\sum _{j=1}^d[A{]}_{ij}[v{]}_{ij}(\log [v{]}_{ij}-1),i=1,\dots ,d.$$

$$\mathcal{L}(\rho ,v;y):={\rho }^{\top }(C(v)+\frac{1}{2}{h}^2)+y{\rho }^{\top }(\tilde{A}(v)h).$$

$${\mathrm{Min}}_{{\pi }_0,u}$$

$$:\frac{\mathrm{d}{\pi }_t}{\mathrm{d}t}={\tilde{A}}^{\top }(u_t){\pi }_t.$$

$$\frac{\mathrm{d}{\pi }_t}{\mathrm{d}t}$$

Full text

11footnotetext: Jin Won Kim and Prashant G. Mehta are with Coordinated Science Laboratory and Department of Mechanical Science and Engineering, University of Illinois at Urbana-Champaign, Urbana, IL 61801 USA

Keywords:Markov Processes, Bayesian Inference, Stochastic Smoothing, Nonlinear Filtering, Duality, Optimal Control

To Michael Dellnitz on the occasion of his 60th birthday

1 Introduction

There is a fundamental dual relationship between estimation and control. The most basic of these relationships is the well known duality between controllability and observability of a linear system [8, Ch. 15]. The relationship suggests that the problem of filter (estimator) design can be re-formulated as a variational problem of optimal control. Such variational formulations are referred to as the duality principle of optimal filtering. The first duality principle appears in the seminal (1961) paper of Kalman-Bucy, where the problem of minimum variance estimation is shown to be dual to a linear quadratic optimal control problem. In these classical settings, the dual variational formulations are of the following two types [1, Sec. 7.3]: (i) minimum variance estimator and (ii) minimum energy estimator.

The classical minimum energy estimator represents a solution of the smoothing problem. The estimator is modeled as a controlled version of the state process in which the process noise term is replaced by a control input. The optimal control input is obtained by maximizing the log of the conditional (smoothed) distribution. For this reason, the estimator is also referred to as the maximum a posteriori (MAP) estimator. The MAP solution coincides with the optimal smoother in the linear-Gaussian case. The earliest construction of the minimum energy estimator is due to Mortensen [11].

A variational formulation of the nonlinear smoothing problem – the focus of this paper – leading to the conditional distribution appears in [10]. The formulation is based upon the variational Kallianpur-Striebel formula [17, Lemma 2.2.1]. The divergence is expressed as an optimal control objective function which turns out to be identical to the objective function considered in the MAP estimator [11]. The difference is that the constraint now is a controlled stochastic process, in contrast to a single trajectory in the MAP estimator. With the optimal control input, the law of the stochastic process is the conditional distribution.

The purpose of this paper is to review and highlight some connections between nonlinear smoothing and optimal control problems involving control of probability densities. In recent years, there has been a lot of interest in mean-field-type optimal control problems where the constraint is a controlled Liouville or a Fokker-Plank equation describing the evolution of the probability density [2, 3, 4]. In this paper, it is shown that the variational formulation proposed in [10] is easily described and solved in these terms. The formulation as a mean-field-type optimal control problem is more natural compared to a stochastic optimal control formulation considered in [10]. In particular, the solution with the density constraint directly leads to the forward-backward equation of pathwise smoothing. This also makes explicit the connection to the log transformation which is known to transform the Bellman equation of optimal control into the Zakai equation of filtering [7, 9]. Apart from the case of the Itô-diffusion, the continuous-time Markov chain is also described. The overall procedure is shown to generalize the classical Mortensen’s minimum energy estimator for the linear Gaussian problem.

The outline of the remainder of this chapter is as follows: the smoothing problem and its solution in terms of the forward-backward Zakai equation and their pathwise representation is reviewed in Sec. 2. The variational formulation leading to a mean-field optimal control problem and its solution appears in Sec. 3. The relationship to the log transformation and to the minimum energy estimator is described. The conclusions appear in Sec. 4. All the proofs are contained in the Appendix.

Notation

We denote the $i^{\text{th}}$ element of a vector by $[\,\cdot\,]_{i}$, and similarly, $(i,j)$ element of a matrix is denoted by $[\,\cdot\,]_{ij}$. $C^{k}(\mathbb{R}^{d}\,;S)$ is the space of functions with continuous $k$-th order derivative. For a function $f\in C^{2}(\mathbb{R}^{d}\,;\mathbb{R})$, $\nabla f$ is the gradient vector and $D^{2}f$ is the Hessian matrix. For a vector field $F\in C^{1}(\mathbb{R}^{d}\,;\mathbb{R}^{d})$, $\operatorname{div}(F)$ denotes the divergence of $F$. For a vector $v\in\mathbb{R}^{d}$, $\operatorname{diag}(v)$ denotes a diagonal matrix with diagonal entries given by the vector; $e^{v}$ and $v^{2}$ are defined in an element-wise manner, that is, $[e^{v}]_{i}=e^{[v]_{i}}$ and $[v^{2}]_{i}=([v]_{i})^{2}$ for $i=1,\ldots,d$. For a matrix, $\mbox{tr}(\cdot)$ denotes the trace.

2 Preliminaries and Background

2.1 The smoothing problem

Consider a pair of continuous-time stochastic processes $(X,Z)$. The state $X=\{X_{t}:t\in[0,T]\}$ is a Markov process taking values in the state space $\mathbb{S}$. The observation process $Z=\{Z_{t}:t\in[0,T]\}$ is defined according to the model:

[TABLE]

where $h:\mathbb{S}\rightarrow\mathbb{R}$ is the observation function and $W=\{W_{t}:t\geq 0\}$ is a standard Wiener process.

The smoothing problem is to compute the posterior distribution ${\sf P}(X_{t}\in\;\cdot\;|{\cal Z}_{T})$ for arbitrary $t\in[0,T]$, where ${\cal Z}_{T}:=\sigma(Z_{s}:0\leq s\leq T)$ is the sigma-field generated by the observation up to the terminal time $T$.

2.2 Solution of the smoothing problem

The smoothing problem requires a model of the Markov process $X$. In applications involving nonlinear smoothing, a common model is the Itô-diffusion in Euclidean settings:

Euclidean state space

The state space $\mathbb{S}=\mathbb{R}^{d}$. The state process $X$ is modeled as an Itô diffusion:

[TABLE]

where $a\in C^{1}(\mathbb{R}^{d};\mathbb{R}^{d})$, $\sigma\in C^{2}(\mathbb{R}^{d};\mathbb{R}^{d\times p})$ and $B=\{B_{t}:t\geq 0\}$ is a standard Wiener process. The initial distribution of $X_{0}$ is denoted as $\nu_{0}(x)\,\mathrm{d}x$ where $\nu_{0}(x)$ is the probability density with respect to the Lebesgue measure. For (1), the observation function $h\in C^{2}(\mathbb{R}^{d};\mathbb{R})$. It is assumed that $X_{0},B,W$ are mutually independent.

The infinitesimal generator of $X$, denoted as ${\cal A}$, acts on $C^{2}$ functions in its domain according to

[TABLE]

The adjoint operator is denoted by ${\cal A}^{\dagger}$. It acts on $C^{2}$ functions in its domain according to

[TABLE]

The solution of the smoothing problem is described by a forward-backward system of stochastic partial differential equations (SPDE) (see [12, Thm. 3.8]):

[TABLE]

where $\overleftarrow{\,\mathrm{d}Z}_{t}$ denotes a backward Itô integral (see [12, Remark 3.3]). The smoothed distribution is then obtained as follows:

[TABLE]

Each of (2) is referred to as the Zakai equation of nonlinear filtering.

2.3 Path-wise representation of the Zakai equations

There is a representation of the forward-backward SPDEs where the only appearance of randomness is in the coefficients. This is referred to as the pathwise (or robust) form of the filter [14, Sec. VI.11].

Using Itô’s formula for $\log p_{t}$,

[TABLE]

Therefore, upon defining $\mu_{t}(x):=\log p_{t}(x)-h(x)Z_{t}$, the forward Zakai equation (2a) is transformed into a parabolic partial differential equation (pde):

[TABLE]

Similarly, upon defining $\lambda_{t}(x)=\log q_{t}(x)+h(x)Z_{t}$, the backward Zakai equation (2b) is transformed into the parabolic pde:

[TABLE]

The pde (3)-(4) are referred to as pathwise equations of nonlinear smoothing.

2.4 The finite state-space case

Apart from Itô-diffusion, another common model is a Markov chain in finite state-space settings:

Finite state space

Let the state-space be $\mathbb{S}=\{e_{1},e_{2},\ldots,e_{d}\}$, the canonical basis in $\mathbb{R}^{d}$. For (1), the linear observation model is chosen without loss of generality: for any function $h:\mathbb{S}\to\mathbb{R}$, we have $h(x)=\tilde{h}^{\top}x$ where $\tilde{h}\in\mathbb{R}^{d}$ is defined by $\tilde{h}_{i}=h(e_{i})$. Thus, the function space on $\mathbb{S}$ is identified with $\mathbb{R}^{d}$. With a slight abuse of notation, we will drop the tilde and simply write $h(x)=h^{\top}x$.

The state process $X$ is a continuous-time Markov chain evolving in $\mathbb{S}$. The initial distribution for $X_{0}$ is denoted as $\nu_{0}$. It is an element of the probability simplex in $\mathbb{R}^{d}$. The generator of the chain is denoted as $A$. It is a $d\times d$ row-stochastic matrix. It acts on a function $f\in\mathbb{R}^{d}$ through right multiplication: $f\mapsto Af$. The adjoint operator is the matrix transpose $A^{\top}$. It is assumed that $X$ and $W$ are mutually independent.

The solution of the smoothing problem for the finite state-space settings is entirely analogous: Simply replace the generator ${\cal A}$ in (2) by the matrix $A$, and the probability density by the probability mass function. The Zakai pde is now the Zakai sde. The formula for the pathwise representation are also entirely analogous:

[TABLE]

with boundary condition $[\mu_{0}]_{i}=\log[\nu_{0}]_{i}$ and $[\lambda_{0}]_{i}=Z_{T}[h]_{i}$, for $i=1,\ldots,d$.

3 Optimal Control Problem

3.1 Variational formulation

For the smoothing problem, an optimal control formulation is derived in the following two steps:

Step 1

A control-modified version of the Markov process $X$ is introduced. The controlled process is denoted as $\tilde{X}:=\{\tilde{X}_{t}:0\leq t\leq T\}$. The control problem is to pick (i) the initial distribution $\pi_{0}\in{\cal P}(\mathbb{S})$ and (ii) the state transition, such that the distribution of $\tilde{X}$ equals the conditional distribution. For this purpose, an optimization problem is formulated in the next step.

Step 2

The optimization problem is formulated on the space of probability laws. Let $P$ denote the law for $X$, $\tilde{P}$ denote the law for $\tilde{X}$, and $Q^{z}$ denote the law for $X$ given an observation path $z=\{z_{t}:0\leq t\leq T\}$. Assuming these are equivalent, the objective function is the relative entropy between $\tilde{P}$ and $Q^{z}$:

[TABLE]

Upon using the Kallianpur-Striebel formula (see [17, Lemma 1.1.5 and Prop. 1.4.2]), the optimization problem is equivalently expressed as follows:

[TABLE]

The first of these terms depends upon the details of the model used to parametrize the controlled Markov process $\tilde{X}$. For the two types of Markov processes, this is discussed in the following sections.

Remark 1

The Schrödinger bridge problem is a closely related problem of recent research interest where one picks $\tilde{P}$ to minimize ${\sf D}(\tilde{P}\|P)$ subject to the constraints on marginals at time $t=0$ and $T$; cf., [5] where connections to stochastic optimal control theory are also described. Applications of such models to the filtering and smoothing problems is discussed in [13]. There are two differences between the Schrödinger bridge problem and the smoothing problem considered here:

The objective function for the smoothing problem also includes an additional integral term in (7) to account for conditioning due to observations $z$ made over time $t\in[0,T]$; 2. 2.

The constraints on the marginals at time $t=0$ and $t=T$ are not present in the smoothing problem. Rather, one is allowed to pick the initial distribution $\pi_{0}$ for the controlled process and there is no constraint present on the distribution at the terminal time $t=T$.

3.2 Optimal control: Euclidean state-space

The modified process $\tilde{X}$ evolves on the state space $\mathbb{R}^{d}$. It is modeled as a controlled Itô-diffusion

[TABLE]

where $\tilde{B}=\{\tilde{B}_{t}:0\leq t\leq T\}$ is a copy of the process noise $B$. The controlled process is parametrized by:

The initial density $\pi_{0}(x)$. 2. 2.

The control function $u\in C^{1}([0,T]\times\mathbb{R}^{d};\mathbb{R}^{p})$. The function of two arguments is denoted as $u_{t}(x)$.

The parameter $\pi_{0}$ and the function $u$ are chosen as a solution of an optimal control problem.

For a given function $v\in C^{1}(\mathbb{R}^{d};\mathbb{R}^{p})$, the generator of the controlled Markov process is denoted by $\tilde{\cal A}(v)$. It acts on a $C^{2}$ function $f$ in its domain according to

[TABLE]

The adjoint operator is denoted by ${\cal A}^{\dagger}(v)$. It acts on $C^{2}$ functions in its domain according to

[TABLE]

For a density $\rho$ and a function $g$, define $\langle\rho,g\rangle:=\int_{\mathbb{R}^{d}}g(x)\rho(x)\,\mathrm{d}x$. With this notation, define the controlled Lagrangian ${\cal L}:C^{2}(\mathbb{R}^{d};\mathbb{R}^{+})\times C^{1}(\mathbb{R}^{d};\mathbb{R}^{p})\times\mathbb{R}\to\mathbb{R}$ as follows:

[TABLE]

The justification of this form of the Lagrangian starting from the relative entropy cost appears in Appendix 5.1.

For a given fixed observation path $z=\{z_{t}:0\leq t\leq T\}$, the optimal control problem is as follows:

[TABLE]

Remark 2

This optimal control problem is a mean-field-type problem on account of the presence of the entropy term ${\sf D}(\pi_{0}\|\nu_{0})$ in the objective function. The Lagrangian is in a standard stochastic control form and the problem can be solved as a stochastic control problem as well [10]. In this paper, the mean-field-type optimal control formulation is stressed as a straightforward way to derive the equations of the nonlinear smoothing.

The solution to this problem is given in the following proposition, whose proof appears in the Appendix 5.3.

Proposition 1

Consider the optimal control problem (8). For this problem, the Hamilton’s equations are as follows:

[TABLE]

The optimal choice of the other boundary condition is as follows:

[TABLE]

where $C=\int_{\mathbb{R}^{d}}\nu_{0}(x)e^{\lambda_{0}(x)}\,\mathrm{d}x$ is the normalization factor. The optimal control is as follows:

[TABLE]

3.3 Optimal control: finite state-space

The modified process $\tilde{X}$ is a Markov chain that also evolves in $\mathbb{S}=\{e_{1},e_{2},\ldots,e_{d}\}$. The control problem is parametrized by the following:

The initial distribution denoted as $\pi_{0}\in\mathbb{R}^{d}$. 2. 2.

The state transition matrix denoted as $\tilde{A}(v)$ where $v\in(\mathbb{R}^{+})^{d\times d}$ is the control input. After [17, Sec. 2.1.1.], it is defined as follows:

[TABLE]

and we set $[v]_{ij}=1$ if $i=j$ or if $[A]_{ij}=0$.

To set up the optimal control problem, define a function $C:(\mathbb{R}^{+})^{d\times d}\to\mathbb{R}^{d}$ as follows

[TABLE]

The Lagrangian for the optimal control problem is as follows:

[TABLE]

The justification of this form of the Lagrangian starting from the relative entropy cost appears in Appendix 5.2.

For given observation path $z=\{z_{t}:0\leq t\leq T\}$, the optimal control problem is as follows:

[TABLE]

The solution to this problem is given in the following proposition, whose proof appears in the Appendix.

Proposition 2

Consider the optimal control problem (10). For this problem, the Hamilton’s equations are as follows:

[TABLE]

The optimal boundary condition for $\pi_{0}$ is given by:

[TABLE]

where $C=\nu_{0}^{\top}e^{\lambda_{0}}$. The optimal control is

[TABLE]

3.4 Derivation of the smoothing equations

The pathwise equations of nonlinear filtering are obtained through a coordinate transformation. The proof for the following proposition is contained in the Appendix 5.5.

Proposition 3

Suppose $(\pi_{t}(x),\lambda_{t}(x))$ is the solution to the Hamilton’s equation (9). Consider the following transformation:

[TABLE]

The pair $(\mu_{t}(x),\lambda_{t}(x))$ satisfy path-wise smoothing equations (3)-(4). Also,

[TABLE]

For the finite state-space case (11), the analogous formulae are as follows:

[TABLE]

and

[TABLE]

for $i=1,\ldots,d$.

3.5 Relationship to the log transformation

In this paper, we have stressed the density control viewpoint. Alternatively, one can express the problem as a stochastic control problem for the $\tilde{X}$ process. For this purpose, define the cost function $l:\mathbb{R}^{d}\times\mathbb{R}^{p}\times\mathbb{R}\rightarrow\mathbb{R}$ as follows:

[TABLE]

The stochastic optimal control problem for the Euclidean case then is as follows:

[TABLE]

Its solution is given in the following proposition whose proof appears in the Appendix 5.6.

Proposition 4

Consider the optimal control problem (12). For this problem, the HJB equation for the value function $V$ is as follows:

[TABLE]

The optimal control is of the state feedback form as follows:

[TABLE]

where $u_{t}(x)=-\sigma^{\top}\nabla(V_{t}+z_{t}h)(x)$.

The HJB equation thus is exactly the Hamilton’s equation (9b) and

[TABLE]

Noting $\lambda_{t}(x)=\log q_{t}(x)+h(x)z_{t}$, the HJB equation for the value function $V_{t}(x)$ is related to the backward Zakai equation for $q_{t}(x)$ through the log transformation (see also [7, Eqn. 1.4]):

[TABLE]

3.6 Linear Gaussian case

The linear-Gaussian case is a special case in the Euclidean setting with the following assumptions on the model:

The drift is linear in $x$. That is,

[TABLE]

where $A\in\mathbb{R}^{d\times d}$ and $H\in\mathbb{R}^{d}$. 2. 2.

The coefficient of the process noise

[TABLE]

is a constant matrix. We denote $Q:=\sigma\sigma^{\top}\in\mathbb{R}^{d\times d}$. 3. 3.

The prior $\nu_{0}$ is a Gaussian distribution with mean $\bar{m}_{0}\in\mathbb{R}^{d}$ and variance $\Sigma_{0}\succ 0$.

For this problem, we make the following restriction: The control input $u_{t}(x)$ is restricted to be constant over $\mathbb{R}^{d}$. That is, the control input is allowed to depend only upon time. With such a restriction, the controlled state evolves according to the sde:

[TABLE]

With a Gaussian prior, the distribution $\pi_{t}$ is also Gaussian whose mean $m_{t}$ and variance $V_{t}$ evolve as follow:

[TABLE]

Since the variance is not affected by control, the only constraint for the optimal control problem is due to the equation for the mean.

It is an easy calculation to see that for the linear model,

[TABLE]

Therefore, the Lagrangian becomes

[TABLE]

provided that $\rho\sim{\cal N}(m,V)$.

For Gaussian distributions $\pi_{0}={\cal N}(m_{0},V_{0})$ and $\nu_{0}={\cal N}(\bar{m}_{0},\Sigma_{0})$, the divergence is given by the well known formula

[TABLE]

and the term due to the terminal condition is easily evaluated as

[TABLE]

Because the control input does not affect the variance process, we retain only the terms with mean and the control and express the optimal control problem as follows:

[TABLE]

By a formal integration by parts,

[TABLE]

This form appears in the construction of the minimum energy estimator [1, Ch. 7.3].

4 Conclusions

In this paper, we provide a self-contained exposition of the equations of nonlinear smoothing as well as connections and interpretations to some of the more recent developments in mean-field-type optimal control theory. These connections suggest that the numerical approaches for mean-field type optimal control problems can also be applied to obtain approximate filters. Development of numerical techniques, e.g., particle filters to empirically approximate the conditional distribution, has been an area of intense research interest; cf., [13] and references therein. Approximate particle filters based upon approximation of dual optimal control-type problems have appeared in [6, 9, 13, 15, 16].

5 Appendix

5.1 Derivation of Lagrangian: Euclidean case

By Girsanov’s theorem, the Radon-Nikodym derivative is obtained (see [13, Eqn. 35]) as follows:

[TABLE]

Thus, we obtain the relative entropy formula:

[TABLE]

5.2 Derivation of Lagrangian: finite state-space case

The derivation of the Lagrangian is entirely analogous to the Euclidean case except the R-N derivative is given according to [17, Prop. 2.1.1]:

[TABLE]

Upon taking log and expectation of both sides, we arrive at the relative entropy formula:

[TABLE]

5.3 Proof of Proposition 1

The standard approach is to incorporate the constraint into the objective function by introducing the Lagrange multiplier $\lambda=\{\lambda_{t}:0\leq t\leq T\}$ as follows:

[TABLE]

Upon using integration by parts and the definition of the adjoint operator, after some manipulation involving completion of squares, we arrive at

[TABLE]

Therefore, it is natural to pick $\lambda$ to satisfy the following partial differential equation:

[TABLE]

with the boundary condition $\lambda_{T}(x)=z_{T}h(x)$. With this choice, the objective function becomes

[TABLE]

which suggest the optimal choice of control is:

[TABLE]

With this choice, the objective function becomes

[TABLE]

which is minimized by choosing

[TABLE]

where $C$ is the normalization constant.

5.4 Proof of Proposition 2

The proof for the finite state-space case is entirely analogous to the proof for the Euclidean case. The Lagrange multiplier $\lambda=\{\lambda_{t}\in\mathbb{R}^{d}:0\leq t\leq T\}$ is introduced to transform the optimization problem into an unconstrained problem:

[TABLE]

Upon using integral by parts,

[TABLE]

The first integrand is

[TABLE]

The minimizer is obtained, element by element, as

[TABLE]

and the corresponding minimum value is obtained by:

[TABLE]

Therefore with the minimum choice of $u_{t}$ above,

[TABLE]

Upon choosing $\lambda$ according to:

[TABLE]

The objective function simplifies to

[TABLE]

where the minimum value is obtained by choosing

[TABLE]

where $C$ is the normalization constant.

5.5 Proof of Proposition 3

Euclidean case

Equation (9b) is identical to the backward path-wise equation (4). So, we need to only derive the equation for $\mu_{t}$. Using the regular form of the product formula,

[TABLE]

With optimal control $u_{t}=\sigma^{\top}\nabla(\lambda_{t}-z_{t}h)$,

[TABLE]

and

[TABLE]

Therefore,

[TABLE]

with the boundary condition $\mu_{0}=\log\nu_{0}$.

Finite state-space case

Equation (11b) is identical to the backward path-wise equation (6). To derive the equation for $\mu_{t}$, use the product formula

[TABLE]

The first term is:

[TABLE]

and the second term is:

[TABLE]

The formula for the optimal control gives

[TABLE]

Combining these expressions,

[TABLE]

which is precisely the path-wise form of the equation (5). At time $t=0$, $\mu_{0}=\log(C[\pi_{0}]_{i})-[\lambda_{0}]_{i}=\log[\nu_{0}]_{i}$.

Smoothing distribution

Since $(\lambda_{t},\mu_{t})$ is the solution to the path-wise form of the Zakai equations, the optimal trajectory

[TABLE]

represents the smoothing distribution.

5.6 Proof of Proposition 4

The dynamic programming equation for the optimal control problem is given by (see [1, Ch. 11.2]):

[TABLE]

Therefore,

[TABLE]

Upon using the completion-of-square trick, the minimum is attained by a feedback form:

[TABLE]

The resulting HJB equation is given by

[TABLE]

with boundary condition $V_{T}(x)=-z_{T}h(x)$. Compare the HJB equation with the equation (14) for $\lambda$, and it follows

[TABLE]