Weak Feller Property of Non-linear Filters

TL;DR

This paper investigates the weak Feller property of non-linear filters in controlled Markov chains, providing new sufficient conditions and alternative proofs that enhance understanding of their stability and approximation capabilities.

Contribution

The paper establishes new conditions for the weak Feller property of non-linear filters, including cases not previously covered, and offers a simplified proof for existing results.

Findings

Proves weak Feller property under weak continuity and total variation continuity of transition probabilities.

Provides an alternative, concise proof for a known result on weak Feller property.

Identifies new conditions under which the weak Feller property holds for non-linear filters.

Abstract

Weak Feller property of controlled and control-free Markov chains lead to many desirable properties. In control-free setups this leads to the existence of invariant probability measures for compact spaces and applicability of numerical approximation methods. For controlled setups, this leads to existence and approximation results for optimal control policies. We know from stochastic control theory that partially observed systems can be converted to fully observed systems by replacing the original state space with a probability measure-valued state space, with the corresponding kernel acting on probability measures known as the non-linear filter (belief) process. Establishing sufficient conditions for the weak Feller property for such processes is a significant problem, studied under various assumptions and setups in the literature. In this paper, we prove the weak Feller property of the non-linear filter process (i) first under weak continuity of the transition probability of controlled Markov chain and total variation continuity of its observation channel, and then, (ii) under total variation continuity of the transition probability of controlled Markov chain. The former result (i) has first appeared in Feinberg et. al. [Math. Oper. Res. 41(2) (2016) 656-681]. Here, we present a concise and easy to follow alternative proof for this existing result. The latter result (ii) establishes weak Feller property of non-linear filter process under conditions, which have not been previously reported in the literature.

Equations

$$I_t=\{Y_{[0,t]},U_{[0,t-1]}\},t\in \text{mathds}N,I_0=\{Y_0\},$$

$$U_t={\gamma }_t(I_t),t\in \text{mathds}{Z}_{+}$$

$$Y_{[0,t]}=\{Y_s,0\le s\le t\},U_{[0,t-1]}=\{U_s,0\le s\le t-1\}.$$

$$\mathop{\rm Pr}\bigl{(}(X_{0},Y_{0})\in B\bigr{)}=\int_{B}Q_{0}(dy_{0}|x_{0})P_{0}(dx_{0}),B\in\mathcal{B}(\mathds{X}\times\mathds{Y}),$$

$$\displaystyle\mathop{\rm Pr}\biggl{(}(X_{t},Y_{t})\in B\,\bigg{|}\,(X,Y,U)_{[0,t-1]}=(x,y,u)_{[0,t-1]}\biggr{)}$$

$$={\int }_{B}Q(d{y}_t|x_t,u_{t-1})\mathcal{T}(d{x}_t|x_{t-1},u_{t-1}),B\in \mathcal{B}(\text{mathds}X\times \text{mathds}Y),$$

$$Z_t(\cdot ):=\mathsf{Pr}\{X_t\in \cdot |Y_0,\dots ,Y_t,U_0,\dots ,U_{t-1}\}\in \mathcal{P}(\text{mathds}X).$$

$$R(B\times C|u_0,z_0)={\int }_{\text{mathds}X}{\int }_{B}Q(C|x_1,u_0)\mathcal{T}(d{x}_1|x_0,u_0)z_0(d{x}_0),$$

$$P(C|u_0,z_0)={\int }_{\text{mathds}X}{\int }_{\text{mathds}X}Q(C|x_1,u_0)\mathcal{T}(d{x}_1|x_0,u_0)z_0(d{x}_0),$$

$$R(B\times C|u_0,z_0)={\int }_{C}F(B|y_1,u_0,z_0)P(d{y}_1|u_0,z_0)$$

$$={\int }_C{z}_1(y_1,u_0,z_0)(B)P(d{y}_1|u_0,z_0),$$

$$\eta (\cdot |z,u)={\int }_{\text{mathds}Y}{1}_{\{F(z,u,y)\in \cdot \}}P(dy|z,u).$$

$$\tilde{c}(z,u):={\int }_{\text{mathds}X}c(x,u)z(dx),$$

$${\tilde{I}}_t=\{Z_{[0,t]},U_{[0,t-1]}\},t\in \text{mathds}N,{\tilde{I}}_0=\{Z_0\}.$$

$${\int }_{\text{mathds}S}f(x){\mu }_n(dx)\to {\int }_{\text{mathds}S}f(x)\mu (dx)$$

$$\displaystyle\rho_{BL}(\mu,\nu)=\sup_{\|f\|_{BL}\leq 1}\biggr{|}\int_{\mathds{S}}f(x)\mu(dx)-\int_{\mathds{S}}f(x)\nu(dx)\biggl{|}$$

$$\displaystyle\rho(\mu,\nu)=\sum_{m=1}^{\infty}2^{-(m+1)}\biggr{|}\int_{\mathds{S}}f_{m}(x)\mu(dx)-\int_{\mathds{S}}f_{m}(x)\nu(dx)\biggl{|},$$

$$\Vert \mu -\nu {\Vert }_{TV}$$

$$X_{t+1}$$

$$Y_t$$

$$\int c(x_1)\mathcal{T}(d{x}_1|x_0^n,u_0^n)=\int c(H(x_0^n,u_0^n,w_0))\mu (d{w}_0)$$

$$\to \int c(H(x_0,u_0,w_0))\mu (d{w}_0)=\int c(x_1)\mathcal{T}(d{x}_1|x_0,u_0),$$

$$Q(dy|x,u)=g(x,u,y)P(dy|z,u)$$

$${\int }_A|g(x_k,u,y)-g(x,u,y)|P(dy|z,u)\to 0.$$

$$\underset{x\in K_n}{\max }\underset{i=1,\dots ,l}{\min }\Vert Q(\cdot |x,u)-{\nu }_i{\Vert }_{TV}<1/3n.$$

$$A_{+}^{(k)}:=\{y\in A:g(x_k,u,y)>g(x,u,y)\},$$

$$A_{-}^{(k)}:=\{y\in A:g(x_k,u,y)<g(x,u,y)\}.$$

$${\int }_A|g(x_k,u,y)-g(x,u,y)|P(dy|z,u)$$

$$={\int }_{A_{+}^{(k)}}g(x_k,u,y)P(dy|z,u)-{\int }_{A_{+}^{(k)}}g(x,u,y)P(dy|z,u)$$

$$+{\int }_{A_{-}^{(k)}}g(x,u,y)P(dy|z,u)-{\int }_{A_{-}^{(k)}}g(x_k,u,y)P(dy|z,u)$$

$$\le |Q(A_{+}^{(k)}|x_k,u)-Q(A_{+}^{(k)}|x,u)|+|Q(A_{-}^{(k)}|x_k,u)-Q(A_{-}^{(k)}|x,u)|$$

$$\le 2\Vert Q(\cdot |x_k,u)-Q(\cdot |x,u){\Vert }_{TV}\to 0.$$

$$\underset{n\to \infty }{\lim }\underset{\lambda \in \Lambda }{\sup }|f_{n,\lambda }(x_n)-f_{\lambda }(x)|=0$$

$$\underset{n\to \infty }{\lim }\underset{\lambda \in \Lambda }{\sup }|f_{\lambda }(x_n)-f_{\lambda }(x)|=0,$$

Full text

††journal: Systems and Control Letterstitletitlefootnotetext: Some of the results in this paper are to be presented at the 2019 IEEE Conference on Decision and Control.

1 Introduction

1.1 Preliminaries

We start with the probabilistic setup of the partially observed Markov processes. Let $\mathds{X}\subset\mathds{R}^{n}$ be a Borel set in which a control-free or controlled Markov process $\{X_{t},\,t\in\mathds{Z}_{+}\}$ takes its value. Here and throughout the paper, $\mathds{Z}_{+}$ denotes the set of non-negative integers and $\mathds{N}$ denotes the set of positive integers. Let $\mathds{U}$, the action space, be a Borel subset of some Euclidean space $\mathds{R}^{p}$. Let $\mathds{Y}\subset\mathds{R}^{m}$ be a Borel set, and let an observation channel $Q$ be defined as a stochastic kernel (regular conditional probability) from $\mathds{X}\times\mathds{U}$ to $\mathds{Y}$ such that $Q(\,\cdot\,|x,u)$ is a probability measure on the Borel $\sigma$-algebra ${\cal B}(\mathds{Y})$ of $\mathds{Y}$ for every $(x,u)\in\mathds{X}\times\mathds{U}$ and $Q(A|\,\cdot\,):\mathds{X}\times\mathds{U}\to[0,1]$ is a Borel measurable function for every $A\in{\cal B}(\mathds{Y})$. Let a decision maker (DM) be located at the output of an observation channel $Q$, with inputs $(X_{t},U_{t-1})$ and outputs $Y_{t}$. An admissible policy $\gamma$ is a sequence of control functions $\{\gamma_{t},\,t\in\mathds{Z}_{+}\}$ such that $\gamma_{t}$ is measurable with respect to the $\sigma$-algebra generated by the information variables

[TABLE]

where

[TABLE]

are the $\mathds{U}$-valued control actions and

[TABLE]

We define $\Gamma$ to be the set of all such admissible policies.

The joint distribution of the state, control, and observation processes is determined by (1) and the following system dynamics:

[TABLE]

where $P_{0}$ is the prior distribution of the initial state $X_{0}$ and $Q_{0}$ is the prior control-free observation channel, and for $t\in\mathds{N}$

[TABLE]

where $\mathcal{T}(\cdot|x,u)$ is a stochastic kernel from $\mathds{X}\times\mathds{U}$ to $\mathds{X}$. This completes the probabilistic setup of the partially observed models.

Remark*.*

Note that if $\mathds{U}=\{u\}$ is a singleton, then we recover the control-free setup. Therefore, results established in this paper also hold for control-free setup.

Often, one is faced with an optimal control problem (or an optimal decision-making problem when control is absent in the transition dynamics). For sake of completeness, we present typical criteria in the following. Let a one-stage cost function $c:\mathds{X}\times\mathds{U}\rightarrow[0,\infty)$, which is a Borel measurable function from $\mathds{X}\times\mathds{U}$ to $[0,\infty)$, be given. Then, we denote by $W(\gamma)$ the cost function of the policy $\gamma\in\Gamma$, which can be, for instance, discounted cost or average cost criteria [1]. With these definitions, the goal of the control problem is to find an optimal policy $\gamma^{*}$ that minimizes $W$.

1.2 Markov Property of Filter Processes

It is known that any POMP can be reduced to a completely observable Markov process [2], [3], whose states are the posterior state distributions or ’beliefs‘ of the observer; that is, the state at time $t$ is

[TABLE]

We call this equivalent process the filter process . The filter process has state space ${\mathds{Z}}={\mathcal{P}}({\mathds{X}})$ and action space ${\mathds{U}}$. Note that ${\mathds{Z}}$ is equipped with the Borel $\sigma$-algebra generated by the topology of weak convergence [4]. Under this topology, ${\mathds{Z}}$ is a Borel space [5]. The transition probability of the filter process can be constructed as follows.

The joint conditional probability on next state and observation variables given the current control action and the current state of the filter process is given by

[TABLE]

for all $B\in{\mathcal{B}}(\mathds{X})$ and $C\in{\mathcal{B}}(\mathds{Y})$. Then, the conditional distribution of the next observation variable given the current state of the filter process and the current control action is given by

[TABLE]

for all $C\in{\mathcal{B}}(\mathds{Y})$. Using this, we can disintegrate $R$ (see [6, Proposition 7.27]) as follows:

[TABLE]

where $F$ is a stochastic kernel from $\mathds{Z}\times\mathds{Y}\times\mathds{U}$ to $\mathds{X}$ and the posterior distribution of $x_{1}$, determined by the kernel $F$, is the state variable $z_{1}$ of the filter process. Then, the transition probability $\eta$ of the filter process can be constructed as follows (see also [7]). If we define the measurable function $F(z,u,y):=F(\,\cdot\,|y,u,z)=\mathop{\rm Pr}\{X_{t+1}\in\,\cdot\,|Z_{t}=z,U_{t}=u,Y_{t+1}=y\}$ from $\mathds{Z}\times\mathds{U}\times\mathds{Y}$ to $\mathds{Z}$ and use the stochastic kernel $P(\,\cdot\,|z,u)=\mathop{\rm Pr}\{Y_{t+1}\in\,\cdot\,|Z_{t}=z,U_{t}=u\}$ from $\mathds{Z}\times\mathds{U}$ to $\mathds{Y}$, we can write $\eta$ as

[TABLE]

The one-stage cost function $\tilde{c}:\mathds{Z}\times\mathds{U}\rightarrow[0,\infty)$ of the filter process is given by

[TABLE]

which is a Borel measurable function. Hence, the filter process is a completely observable Markov process with the components $({\mathds{Z}},{\mathds{U}},\tilde{c},\eta)$.

For the filter process, the information variables is defined as

[TABLE]

It is a standard result that an optimal control policy of the original POMP will use the belief $Z_{t}$ as a sufficient statistic for optimal policies (see [2], [3]). More precisely, the filter process is equivalent to the original POMP in the sense that for any optimal policy for the filter process, one can construct a policy for the original POMP which is optimal.

1.3 Problem Formulation

Let us review two convergence notions for probability measures, and also, we define the weak Feller property of Markov decision processes. Let $(\mathds{S},d)$ be a separable metric space. A sequence $\{\mu_{n},n\in\mathds{N}\}$ in the set of probability measures $\mathcal{P}(\mathds{S})$ is said to converge to $\mu\in\mathcal{P}(\mathds{S})$ weakly if

[TABLE]

for every continuous and bounded $f:\mathds{S}\to\mathds{R}$. The topology of weak convergence on the set of probability measures on a separable metric space is metrizable. One such metric is the bounded-Lipschitz metric. For any two probability measures $\mu$ and $\nu$, the bounded-Lipschitz metric is defined as:

[TABLE]

where $\|f\|_{BL}=\|f\|_{\infty}+\sup_{x\neq y}\frac{|f(x)-f(y)|}{d(x,y)}$ and $\|f\|_{\infty}=\sup_{x\in\mathds{S}}|f(x)|$. Another metric that metrizes the weak topology on ${\mathcal{P}}(\mathds{S})$ is the following:

[TABLE]

where $\{f_{m}\}_{m\geq 1}$ is an appropriate sequence of continuous and bounded functions such that $\|f_{m}\|_{\infty}\leq 1$ for all $m\geq 1$ (see [5, Theorem 6.6, p. 47]).

For probability measures $\mu,\nu\in\mathcal{P}(\mathds{S})$, the total variation norm is given by

[TABLE]

where the supremum is taken over all Borel measurable real $f$ such that $\|f\|_{\infty}\leq 1$. A sequence $\{\mu_{n},n\in\mathds{N}\}$ in $\mathcal{P}(\mathds{S})$ is said to converge in total variation to $\mu\in\mathcal{P}(\mathds{S})$ if $\|\mu_{n}-\mu\|_{TV}\to 0$.

Definition 1** (Weak Feller Property).**

We say that a Markov decision process with transition kernel $\mathcal{T}(\cdot|x,u)$ has weak Feller property if $\mathcal{T}$ is weakly continuous in $x$ and $u$; that is, if $(x_{n},u_{n})\rightarrow(x,u)$, then $\mathcal{T}(\cdot|x_{n},u_{n})\rightarrow\mathcal{T}(\cdot|x,u)$ weakly.

With this definition, we can now state the problem that we are interested in this paper.

• (P)

Under what conditions on the transition kernel and the observation channel of the partially observed model, the filter process has a weak Feller property?

1.4 Significance of the Problem

In this section, we motivate the operational (in view of engineering applications) and the mathematical importance of the problem in view of existence and invariance properties, approximations and computational results involving non-linear filters and stochastic control, and related applications involving particle filtering.

For finite-horizon problems and a large class of infinite-horizon discounted cost problems, it is a standard result that an optimal control policy will use the filter process as a sufficient statistic for optimal policies (see [2, 3, 8]). Hence, the partially observed model and the corresponding filter process are equivalent in the sense of cost minimization. Therefore, results developed for the standard controlled Markov process problems (e.g., measurable selection criteria as summarized in [1, Chapter 3]) can be applied to the filter processes, and so, to the partially observed models. In controlled Markov processes theory, weak continuity of the transition kernel is an important condition leading to both the existence of optimal control policies for finite-horizon and infinite-horizon discounted cost problems as well as the continuity properties of the value function (see, e.g., [9, Section 8.5]).

For partially observed stochastic control problems with the average cost criterion, the conditions of existence of optimal policies stated in the literature are somewhat restrictive, with the most relaxed conditions to date being reported in [10, 11], to our knowledge. For such average cost stochastic control problems, the weak Feller property can be utilized to obtain a direct method to establish the existence of optimal stationary (and possibly randomized) control policies. Indeed, for such problems, the convex analytic method ([12] and [13]) is a powerful approach to establish the existence of optimal policies. If one can establish the weak Feller property of the filter process, then the continuity and compactness conditions utilized in the convex program of [13] would lead to the existence of optimal average cost policies.

In addition to existence of optimal policies, the weak Feller property has also recently been shown to lead to (asymptotic) consistency in approximation methods for controlled Markov processes with uncountable state and action spaces. In [14, 15], authors showed that optimal policies obtained from finite-model approximations to infinite-horizon discounted cost controlled Markov processes with Borel state and action spaces asymptotically achieve the optimal cost for the original problem under weak Feller property. Hence, the weak Feller property of filter process suggests that approximation results in [14, 15], which only require weak continuity conditions on the transition probability of a given controlled Markov model, are particularly suitable in developing approximation methods for partially observed models (through their reduction to fully observed models).

For control-free setups, the weak Feller property of the filter processes leads to the existence of invariant probability measures when the hidden Markov processes take their values in compact spaces or more general spaces under appropriate tightness/regularity conditions [16, 17, 33].

For the empirical consistency and convergence results involving the very popular particle filtering algorithms, weak Feller property of the filter processes is a commonly imposed, implicit, assumption (see e.g. [18, 19]). Finally, for the study of ergodicity and asymptotic stability of nonlinear filters, weak Feller property also plays an important role (see [20, 21, 17]).

2 Main Results and Connections with the Literature

2.1 Statement of Main Results

In this paper, we show the weak Feller property of the filter process under two different set of assumptions.

Assumption 1*.*

• (i)

The transition probability $\mathcal{T}(\cdot|x,u)$ is weakly continuous in $(x,u)$, i.e., for any $(x_{n},u_{n})\to(x,u)$, $\mathcal{T}(\cdot|x_{n},u_{n})\to\mathcal{T}(\cdot|x,u)$ weakly.

• (ii)

The observation channel $Q(\cdot|x,u)$ is continuous in total variation, i.e., for any $(x_{n},u_{n})\to(x,u)$, $Q(\cdot|x_{n},u_{n})\rightarrow Q(\cdot|x,u)$ in total variation.

Assumption 2*.*

• (i)

The transition probability $\mathcal{T}(\cdot|x,u)$ is continuous in total variation in $(x,u)$, i.e., for any $(x_{n},u_{n})\to(x,u)$, $\mathcal{T}(\cdot|x_{n},u_{n})\to\mathcal{T}(\cdot|x,u)$ in total variation.

• (ii)

The observation channel $Q(\cdot|x)$ is independent of the control variable.

We now formally state the main results of our paper.

Theorem 1* (Feinberg et. al. [26]).*

Under Assumption 1, the transition probability $\eta(\cdot|z,u)$ of the filter process is weakly continuous in $(z,u)$.

Theorem 2*.*

Under Assumption 2, the transition probability $\eta(\cdot|z,u)$ of the filter process is weakly continuous in $(z,u)$.

Theorem 1 is originally due to Feinberg et. al. [26]. The contribution of our paper is that the proof presented here is more direct and significantly more concise. Theorem 2 establishes that under Assumption 2 (with no assumptions on the measurement model), the filter process is weakly continuous. This result has not been previously reported in the literature.

The proofs of these results are presented in Section 3.

Remark*.*

We note that there are examples where Assumptions Assumption 1 or 2 are not satisfied (e.g. [26, Section 8.2]), however the filter kernel $\eta$ is still weak Feller. We refer the reader to the result [26, Theorem 7.1] which establishes weak Feller property of the filter kernel under further sets of assumptions. We also note that, after the first submission of this paper to Arxiv, [28, Theorem 4.4] reported a result generalizing Theorem 2 to the channel $Q(\cdot|x,u)$ depending on the control variable as well and being continuous in total variation in $u$.

2.2 Examples

In this section, we give concrete examples for the system and observation channel models which satisfy Assumption 1 or Assumption 2. Suppose that the system dynamics and the observation channel are represented as follows:

[TABLE]

where $W_{t}$ and $V_{t}$ are i.i.d. noise processes. This is, without loss of generality, always the case; that is, one can transform the dynamics of any partially observed model into this form (see Lemma 1.2 in [31], or Lemma 3.1 of [32]).

• (i)

Suppose that $H(x,u,w)$ is a continuous function in $x$ and $u$. Then, the corresponding transition kernel is weakly continuous. To see this, observe that, for any $c\in C_{b}(\mathds{X})$, we have

[TABLE]

where we use $\mu$ to denote the probability model of the noise.

• (ii)

Suppose that $G(x,u,v)=g(x,u)+v$, where $g$ is a continuous function and $V_{t}$ admits a continuous density function $\varphi$ with respect to some reference measure $\nu$. Then, the channel is continuous in total variation. Notice that under this setup, we can write $Q(dy|x,u)=\varphi(y-h(x,u))\nu(dy)$. Hence, the density of $Q(dy|x_{n},u_{n})$ converges to the density of $Q(dy|x,u)$ pointwise, and so, $Q(dy|x_{n},u_{n})$ converges to $Q(dy|x,u)$ in total variation by Scheffé’s Lemma [22, Theorem 16.12]. Hence, $Q(dy|x,u)$ is continuous in total variation under these conditions.

• (iii)

Suppose that we have $H(x,u,w)=h(x,u)+w$, where $f$ is continuous and $W_{t}$ admits a continuous density function $\varphi$ with respect to some reference measure $\nu$. Then, the transition probability is continuous in total variation. Again, notice that with this setup we have $\mathcal{T}(dx_{1}|x_{0},u_{0})=\varphi(x_{1}-h(x_{0},u_{0}))\nu(dx_{1})$. Thus, continuity of $\varphi$ and $h$ guarantees the pointwise convergence of the densities, so we can conclude that the transition probability is continuous in total variation by again Scheffé’s Lemma.

The analysis in the paper will provide weak Feller results for a large class of partially observed control systems as reviewed in the aforementioned examples. In particular, if the state dynamics are affected by an additive noise process which admits a continuous density, we can guarantee weak Feller property of the filter process by means of Theorem 2 without referring to the noise model of the observation channel.

2.3 Comparison with the Prior Results

Weak Feller property of the control-free transition probability of the filter processes has been established using different approaches and different conditions. In [21] it has been shown that, for continuous-time filter processes, if the signal process (state process of the partially observed model) is weak Feller and the measurement channel is an additive channel in the form $Y_{t}=\int_{0}^{t}h(X_{u})du+V_{t}$, where $h$ is assumed to be continuous and possibly unbounded and $V_{t}$ is a standard Wiener process, then the filter process itself is also weak Feller. In [17], the authors study the discrete-time filter processes, where the state process noise may not be independent of the observation process noise; it has been shown that if the observation model is additive in the form $Y_{t}=h(X_{t})+V_{t}$, where $h$ is assumed to be continuous and $V_{t}$ is an i.i.d. noise process which admits a continuous and bounded density function, then the observation and filter processes $(Y_{t},Z_{t})$ are jointly weak Feller. In [20], the weak Feller property of the filter process has been shown for both discrete and continuous time setups when the channel is additive, $Y_{t}=h(X_{t})+V_{t}$, where $h$ is bounded and continuous and $V_{t}$ is an i.i.d. noise process with a continuous, bounded and positive density function.

[18, 19] have studied the consistency of the particle filter methods where the weak Feller property of the filter process has been used to establish the convergence results. In [18], it has been shown that the filter process is weak Feller under the assumption that the transition probability of the partially observed system is weak Feller and the measurement channel is an additive channel in the form $Y_{t}=h(X_{t})+V_{t}$, where $h$ is assumed to be continuous and $V_{t}$ is an i.i.d. noise process, which admits a continuous and bounded density function with respect to the Lebesgue measure. In [19], the weak Feller property of the filter process has been established under the assumption that the measurement channel admits a continuous and bounded density function with respect to the Lebesgue measure; i.e., the channel can be written in the following form: $Q(y\in A|x)=\int_{A}g(x,y)dy$ for any$A\in{\mathcal{B}}(\mathds{Y})$ and for any $x\in\mathds{X}$, where $g$ is a continuous and bounded function.

Weak Feller property of the controlled transition probability of the filter processes has been established, in the most general case to date, by Feinberg et.al. [26]. Under the assumption that the measurement channel is continuous in total variation and the transition kernel of the partially observed model is weak Feller, the authors have established the weak Feller property of the transition probability of the filter process. In Section 4, we will give a detailed discussion on the methods used by Feinberg et.al. [26], and also, we will compare their approach with ours.

As reviewed above, the prior literature often used the additive channel model $Y_{t}=h(X_{t},U_{t-1})+V_{t}$ with various regularity assumptions on $h$ and the noise model $V_{t}$. When the observation channel is additive $Y_{t}=h(X_{t},U_{t-1})+V_{t}$, where $h$ is continuous and $V_{t}$ admits a continuous density function with respect to some measure $\mu$, one can show that the channel also admits a continuous density function, i.e., $Q(y\in A|x,u)=\int_{A}g(x,u,y)\mu(dy)$ for any $A\in{\mathcal{B}}(\mathds{Y})$ and for any $(x,u)\in\mathds{X}\times\mathds{U}$. When the observation channel has a continuous density function, the pointwise convergence of the density functions implies the total variation convergence by Scheffé’s Lemma [22, Theorem 16.12]. Thus, $g(x_{k},u_{k},y)\to g(x,u,k)$ for some $(x_{k},u_{k})\to(x,u)$ implies that $Q(\cdot|x_{k},u_{k})\to Q(\cdot|x,u)$ in total variation, i.e., the observation channel is continuous in total variation.

In the following, we develop a relationship between the total variation continuity of the channel (as required by [26] and in our Theorem 1) and the more restrictive density conditions on the measurement channels presented in the prior works [21, 17, 19, 18].

In the theorem below, we show that having a continuous density is almost equivalent to the condition that the observation channel is continuous in total variation. To our knowledge, it is the first result in the literature making the connection between channels which are continuous in total variation and channels which admit a density function.

Theorem 3*.*

Suppose that the observation channel $Q(dy|x,u)$ is continuous in total variation. Then, for any $(z,u)\in\mathds{Z}\times\mathds{U}$, we have, $\mathcal{T}(\cdot|z,u)$-a.s., that $Q(dy|x,u)\ll P(dy|u,z)$ and

[TABLE]

for a measurable function $g$, which satisfies for any $A\in{\mathcal{B}}(\mathds{Y})$ and for any $x_{k}\to x$

[TABLE]

Proof.

Fix any $(z,u)$. We first show that $Q(dy|x,u)\ll P(dy|z,u)$, $\mathcal{T}(\cdot|z,u)$-a.s.. Note that $Q(dy|x,u)\ll P(dy|z,u)$ if and only if, for all $\varepsilon>0$, there exits $\delta>0$ such that $Q(A|x,u)<\varepsilon$ whenever $P(A|z,u)<\delta$. For each $n\geq 1$, let $K_{n}\subset\mathds{X}$ be compact such that $\mathcal{T}(K_{n}|z,u)>1-/3n$. As $Q(dy|x,u)$ is continuous in total variation norm, the image of $K_{n}\times\{u\}$ under $Q(dy|x,u)$ is compact in ${\mathcal{P}}(\mathds{Y})$. Hence, there exist $\{\nu_{1},\ldots,\nu_{l}\}\subset{\mathcal{P}}(\mathds{Y})$ such that

[TABLE]

Define the following stochastic kernel $\nu_{n}(\cdot|x,u)=\arg\min_{\nu_{i}}\|Q(\cdot|x,u)-\nu_{i}\|_{TV}$. Then, we define $P_{n}(\cdot|z,u)=\int_{\mathds{X}}\nu_{n}(\cdot|x,u)\mathcal{T}(dx|z,u)$. One can prove that $\|P(\cdot|z,u)-P_{n}(\cdot|z,u)\|_{TV}<1/n$. Moreover, since $P_{n}(\cdot|z,u)$ is a mixture of finite probability measures $\{\nu_{1},\ldots,\nu_{l}\}$, we have that $\nu_{n}(\cdot|x,u)\ll P_{n}(\cdot|z,u)$ for all $x\in C_{n}$, where $\mathcal{T}(C_{n}|z,u)=1$. Let $C=\bigcap_{n}C_{n}$, and so, $\mathcal{T}(C|z,u)=1$. We claim that if $x\in C$, then $Q(dy|x,u)\ll P(dy|z,u)$, which completes the proof of the first statement. To prove the claim, fix any $\varepsilon>0$ and choose $n\geq 1$ such that $\varepsilon>2/3n$. Then, there exists $\delta>0$ such that $\nu_{n}(A|x,u)<\varepsilon/2$ whenever $P_{n}(A|z,u)<\delta$. This implies that $Q(A|x,u)<\varepsilon$ whenever $P(A|z,u)<\delta+1/n$. Hence, $Q(dy|x,u)\ll P(dy|z,u)$.

To show the second claim, for any $A\in{\mathcal{B}}(\mathds{Y})$ and for any $x_{k}\to x$ , we define

[TABLE]

[TABLE]

With these sets, we have

[TABLE]

∎

We again emphasize that weak Feller property of the filter process under Assumption 1 has been first established by [26] using different method compared to ours. Our method is significantly more concise and direct. It is also important to note that Assumption 2 completely eliminates any restriction on the observation channel to establish the weak Feller property of filter process. This relaxation is quite important in practice since modelling the noise on the observation channel in control problems is quite cumbersome, and in general, infeasible. But, in many problems that arise in practice, the transition probability has a continuous density with respect to some reference measure, which directly implies, via Scheffé’s Lemma, the total variation continuity of the transition probability. We also note that the weak Feller property under only Assumption 2-(i) cannot be established. Indeed, Example 4.1 of [26] shows that the total variation continuity assumption on the observation channel cannot be relaxed even when the transition kernel is continuous in total variation to prove weak Feller property of the filter process under a controlled observation channel model: A careful look at the counterexample shows that it indeed uses the discontinuity of the observation channel in the control action to prove that the filter process cannot be weak Feller when the observation channel is not continuous in total variation and the transition kernel is continuous in total variation.

3 Proofs

The following result will play a key role for the proofs of main results. The proof can be found on the appendix.

Lemma 1*.*

Let $\mathds{X}$ be a Borel space. Suppose that we have a family of uniformly bounded real Borel measurable functions $\{f_{n,\lambda}\}_{n\geq 1,\lambda\in\Lambda}$ and $\{f_{\lambda}\}_{\lambda\in\Lambda}$, for some set $\Lambda$. If, for any $x_{n}\to x$ in $\mathds{X}$, we have

[TABLE]

[TABLE]

then, for any $\mu_{n}\to\mu$ weakly in ${\mathcal{P}}(\mathds{X})$, we have

[TABLE]

In Theorem 1 and Theorem 2, we need to show that, for every $(z_{0}^{n},u_{n})\to(z_{0},u)$ in $\mathds{Z}\times\mathds{U}$, we have

[TABLE]

where we equip $\mathds{Z}$ with the metric $\rho$ to define bounded-Lipschitz norm $\|f\|_{BL}$ of any Borel measurable function $f:\mathds{Z}\rightarrow\mathds{R}$. We can equivalently write this as

[TABLE]

The term in equation (7) can be upper bounded as follows:

[TABLE]

where, in the last inequality, we have used $\|f\|_{\infty}\leq\|f\|_{BL}\leq 1$. To prove that (3) (and so (7)) goes to [math], it is sufficient to establish the following results:

• (i)

$P(dy_{1}|z_{0},u_{0})$ is continuous in total variation,

• (ii)

$\lim_{n\rightarrow\infty}\int_{\mathds{Y}}\rho(z_{1}(z_{0}^{n},u_{n},y_{1}),z_{1}(z_{0},u,y_{1}))P(dy_{1}|z_{0},u)=0$ as $(z_{0}^{n},u_{n})\rightarrow(z_{0},u)$.

Indeed, suppose that (i) and (ii) hold. Then, the first term in (3) goes to [math] as $P(\cdot|z_{0},u)$ is continuous in total variation. For the second term in (3), we have

[TABLE]

Therefore, to complete the proof of Theorem 1 and Theorem 2, we will prove (i) and (ii).

3.1 Proof of Theorem 1

We first prove (i); that is, $P(dy_{1}|z_{0},u)$ is continuous in total variation. To this end, let $(z_{0}^{n},u_{n})\rightarrow(z_{0},u)$. Then, we write

[TABLE]

where $\mathcal{T}(dx_{1}|z_{0}^{n},u_{n})\coloneqq\int_{\mathds{X}}\mathcal{T}(dx_{1}|x_{0},u_{n})z_{0}^{n}(dx_{0})$. Note that, by Lemma 1, we can show that $\mathcal{T}(dx_{1}|z_{0}^{n},u_{n})\to\mathcal{T}(dx_{1}|z_{0},u)$ weakly. Indeed, if $g\in C_{b}(\mathds{X})$, then we define $r_{n}(x_{0})=\int_{\mathds{X}}g(x_{1})\mathcal{T}(dx_{1}|x_{0},u_{n})$ and $r(x_{0})=\int_{\mathds{X}}g(x_{1})\mathcal{T}(dx_{1}|x_{0},u)$. Since $\mathcal{T}(dx_{1}|x_{0},u)$ is weakly continuous, we have $r_{n}(x_{0}^{n})\rightarrow r(x_{0})$ when $x_{0}^{n}\rightarrow x_{0}$. Hence, by Lemma 1, we have

[TABLE]

Hence, $\mathcal{T}(dx_{1}|z_{0}^{n},u_{n})\to\mathcal{T}(dx_{1}|z_{0},u)$ weakly. Moreover, the families of functions $\{Q(A|\,\cdot\,,u_{n})\}_{n\geq 1,A\in{\mathcal{B}}(\mathds{Y})}$ and $\{Q(A|\,\cdot\,,u)\}_{A\in{\mathcal{B}}(\mathds{Y})}$ satisfy the conditions of Lemma 1 as $Q$ is continuous in total variation distance. Therefore, Lemma 1 yields that

[TABLE]

Thus, $P(dy_{1}|z_{0},u)$ is continuous in total variation.

To prove (ii), we write

[TABLE]

where we have used Fubini’s theorem with the fact that $\sup_{m}\|f_{m}\|_{\infty}\leq 1$. For each $m$, let us define

[TABLE]

Then, we can write

[TABLE]

In the sequel, we only consider the term with the set $I_{+}^{(n)}$. The analysis for the other one follows from the same steps. We have

[TABLE]

where we have used $\|f_{m}\|_{\infty}\leq 1$ in the last inequality. The first term above goes to [math] since $P(dy_{1}|z_{0},u)$ is continuous in total variation. For the second term, we use Lemma 1. Indeed, families of functions $\{f_{m}(\cdot)Q(A|\,\cdot\,,u_{n}):n\geq 1,A\in{\mathcal{B}}(\mathds{Y})\}$ and $\{f_{m}(\cdot)Q(A|\,\cdot\,,u):A\in{\mathcal{B}}(\mathds{Y})\}$ satisfy the conditions in Lemma 1 as $Q$ is continuous in total variation. Hence, the second term converges to [math] by Lemma 1 since $\mathcal{T}(dx_{1}|z_{0}^{n},u_{n})\rightarrow\mathcal{T}(dx_{1}|z_{0},u)$ weakly. Hence, for each $m$, we have

[TABLE]

By the dominated convergence theorem, we then have

[TABLE]

This establishes (ii), which completes the proof together with (i).

3.2 Proof of Theorem 2

We first show (i); that is, $P(dy_{1}|z_{0},u_{0})$ is continuous total variation. Let $(z_{0}^{n},u_{n})\rightarrow(z_{0},u)$. Then, we have

[TABLE]

For each $A\in{\mathcal{B}}(\mathds{Y})$ and $n\geq 1$, we define

[TABLE]

and

[TABLE]

Then, for all $x_{0}^{n}\to x_{0}$, we have

[TABLE]

Then, by Lemma 1, we have

[TABLE]

Hence, $P(dy_{1}|z_{0},u_{0})$ is continuous in total variation.

Now, we show (ii); that is, for any $(z_{0}^{n},u_{n})\rightarrow(z_{0},u)$, we have

[TABLE]

From the proof of Theorem 1, it suffices to show that

[TABLE]

Indeed, we have

[TABLE]

where we have used \sup_{n\geq 1}\sup_{x_{1}\in\mathds{X}}\big{|}f_{m}(x_{1})Q(I_{+}^{(n)}|x_{1})\big{|}\leq 1 in the last inequality. If we define $r_{n}(x_{0})=\|\mathcal{T}(dx_{1}|x_{0},u_{n})-\mathcal{T}(dx_{1}|x_{0},u)\|_{TV}$, then $r_{n}(x_{0}^{n})\rightarrow 0$ whenever $x_{0}^{n}\rightarrow x_{0}$. Then, the first term converges to [math] by Lemma 1 as $z_{0}^{n}\rightarrow z_{0}$ weakly. The second term also converges to [math] by Lemma 1, since $\{\int_{\mathds{X}}f(x_{1})Q(I_{+}^{(n)}|x_{1})\mathcal{T}(dx_{1}|\cdot,u):n\geq 1\}$ is a family of uniformly bounded and equicontinuous functions by total variation continuity of $\mathcal{T}(dx_{1}|x_{0},u)$. This proves (ii) and completes the proof together with (i).

4 A Technical Generalization

In this section, we prove the weak Feller property of the filter process under more general condition than those in Theorem 1 and Theorem 2. But, we note that it is indeed generally infeasible to establish this condition without imposing assumptions similar to the assumptions in Theorem 1 and Theorem 2. Therefore, although this condition is more general than those in Theorem 1 and Theorem 2, this generalization is not excessively important in practice.

We first note that our proof technique brings to light the main ingredients that is necessary to prove the weak Feller property of the filter process via the item (i) and eq. (10); that is,

$P(dy_{1}|z_{0},u_{0})$ is continuous in total variation,

• 2.

$\displaystyle\lim_{n\rightarrow\infty}\int_{\mathds{X}}f_{m}(x_{1})Q(I_{+}^{(n)}|x_{1})\mathcal{T}(dx_{1}|z_{0}^{n},u_{n})-\int_{\mathds{X}}f_{m}(x_{1})Q(I_{+}^{(n)}|x_{1})\mathcal{T}(dx_{1}|z_{0},u)=0.$

This observation suggests the following condition that generalize the conditions in our previously stated main results. Let $\mathds{F}=\{f_{m}\}_{m\geq 1}\subset C_{b}(\mathds{X})$ be a countable set of continuous and bounded functions such that $\|f_{m}\|_{\infty}\leq 1$ for all $m\geq 1$, $1_{\mathds{X}}\in\mathds{F}$, and $\mathds{F}$ metrizes the weak topology on $\mathcal{P}(\mathds{X})$ via the metric $\rho$ introduced in Section 1.3. Then, we state the following assumption:

• (M)

For each $f\in\mathds{F}$, the family of functions

[TABLE]

is equicontinuous when indexed by $A\in{\mathcal{B}}(\mathds{Y})$.

Using Lemma 1, it is fairly straightforward to prove that conditions in Theorem 1 and Theorem 2 both imply the assumption (M). Hence, assumption (M) is more general than those in Theorem 1 and Theorem 2.

Theorem 4*.*

Under assumption (M), the transition probability $\eta(\cdot|z,u)$ of the filter process is weakly continuous in $(z,u)$.

Proof.

Recall that it is sufficient to prove the following:

• (i)

$P(dy_{1}|z_{0},u_{0})$ is continuous in total variation,

• (ii)

$\lim_{n\rightarrow\infty}\int_{\mathds{Y}}\rho(z_{1}(z_{0}^{n},u_{n},y_{1}),z_{1}(z_{0},u,y_{1}))P(dy_{1}|z_{0},u)=0$ as $(z_{0}^{n},u_{n})\rightarrow(z_{0},u)$.

Firstly, (i) is true since $1_{\mathds{X}}\in\mathds{F}$. For (ii), it suffices to show that

[TABLE]

But this immediately follows from assumption (M). ∎

A careful look at the proof of the weak Feller property of the filter process in Feinberg et. al. [26] reveals that they have first established the weak Feller property under a condition somewhat similar to the assumption (M), and then, establish Theorem 1 by proving that assumptions in Theorem 1 imply this more general condition. Indeed, let $\tau_{b}=\{O_{j}\}\subset\mathds{X}$ be a countable base for the topology on $\mathds{X}$ such that $\mathds{X}\in\tau_{b}$. Then, under the following assumption:

• (F)

For each finite intersection $O=\bigcap_{n=1}^{N}O_{j_{n}}$, where $O_{j_{n}}\in\tau_{b}$, the family of functions

[TABLE]

is equicontinuous when indexed by $A\in{\mathcal{B}}(\mathds{Y})$,

they have proved that the weak Feller property of the filter process holds (see [26, Lemma 5.3] and [25, Theorem 5.5]). We observe that (F) is very similar to (M) except that, in (F), Feinberg et. al. use open sets instead of continuous and bounded functions. However, proving that conditions in Theorem 1 imply the assumption (F) as in [26] requires quite tedious mathematical methods. By using open sets instead of continuous and bounded functions, one needs to work with inequalities and limit infimum operation as a result of Portmanteau theorem [4, Theorem 2.1] (and the associated proof program involving generalized Fatou’s lemma [24, 27]), in place of equalities and limit operation, which are significantly easier to analyze than the former leading to a much more concise analysis that we have presented in this paper. For instance, [26, Theorem 5.1] is the key result to prove that the weak Feller condition of the transition probability and the total variation continuity of the observation channel imply the assumption (F). We note that if one states this result using continuous and bounded functions in place of open sets, then this version of [26, Theorem 5.1] becomes a corollary of Lemma 1, which has a concise and easy to follow proof. But, the proof of [26, Theorem 5.1] with open sets requires quite tedious mathematical concepts from topology and weak convergence of probability measures. In view of this discussion, we also note that Theorem 2 can also be proved using the condition (F) rather than our approach building on (M) through some additional argumentation.

In summary, our approach allows for a more direct and concise approach which also makes the proof of Theorem 1 more accessible. Once again, we note that Theorem 2 has not been reported in the literature.

5 Conclusion

In this paper, there are two main contributions: (i) the weak Feller property of the filter process is established under a new condition, which assumes that the state transition probability is continuous under the total variation convergence with no assumptions on the measurement model, and (ii) a concise and easy to follow proof of the same result under the weak Feller condition of the transition probability and the total variation continuity of the observation channel, which was first established in [26], is also given. Implications of these results have also been presented.

6 Acknowledgements

The authors are grateful to Prof. Eugene Feinberg for generously providing technical feedback and pointing out to related results.

Appendix

6.1 Proof of Lemma 1

Note that since, for any $x_{n}\to x$ in $\mathds{X}$, we have

[TABLE]

we see that $\{f_{\lambda}\}_{\lambda\in\Lambda}$ is an equicontinuous family of functions. Thus, by the Arzela-Ascoli Theorem [29], for any given compact set $K\subset\mathds{X}$ and $\epsilon>0$, there is a finite set of continuous and bounded functions $\mathds{F}:=\{f_{1},\dots,f_{N}\}$, so that, for any $\lambda\in\Lambda$, there is $f_{i}\in\mathds{F}$ with

[TABLE]

Now, we claim that, for the same $f_{i}\in\mathds{F}$, we have $\sup_{x\in K}|f_{n,\lambda}(x)-f_{i}(x)|\leq 3\epsilon/2$ for large enough $n$, which is independent of $\lambda$. To see this, observe the following:

[TABLE]

Note that the second term is less than $\epsilon$ and the first term can be made arbitrarily small as $f_{n,\lambda}\to f_{\lambda}$ uniformly on compact sets and on $\Lambda$, which can be easily proved using the assumptions in the lemma.

Note that $\mu_{n}\to\mu$ weakly. Hence, $\{\mu_{n}\}$ is a tight family of probability measures by Prokhorov theorem [30, Theorem 5.2]. Therefore, for any $\epsilon>0$, there exists a compact subset $K_{\epsilon}$ of $\mathds{X}$ such that, for all $n$,

[TABLE]

Now, we fix any $\epsilon>0$ and choose a compact set $K_{\epsilon}$ such that, for all $n$, $\mu_{n}(K_{\epsilon})\geq 1-\epsilon$. We also fix a finite family of continuous and bounded functions $\mathds{F}:=\{f_{1},\dots,f_{N}\}$ such that, for any $\lambda$, we can find $f_{i}\in\mathds{F}$ with $\sup_{x\in K_{\epsilon}}|f_{\lambda}(x)-f_{i}(x)|\leq\epsilon$. Moreover, we choose a large $N$ such that $\sup_{x\in K_{\epsilon}}|f_{n,\gamma}(x)-f_{i}(x)|\leq 3\epsilon/2$ for all $n\geq N$.

With this setup, we go back to the main statement:

[TABLE]

where $C$ is the uniform bound on $\{f_{n,\lambda}\}$ and $\{f_{\lambda}\}$. Since $\epsilon$ is arbitrary and $\mu_{n}$ converges weakly to $\mu$, the result follows.