Modeling and Analysis of Switching Diffusion Systems: Past-Dependent Switching with a Countable State Space

TL;DR

This paper develops a mathematical framework for analyzing complex hybrid stochastic systems with countably infinite discrete states and past-dependent switching, establishing fundamental properties like existence, uniqueness, and Feller properties.

Contribution

It introduces a novel class of switching diffusion models with countable state spaces and past-dependent switching, proving key theoretical properties for these systems.

Findings

Existence and uniqueness of solutions established.

Markov and Feller properties proved for the hybrid process.

Strong Feller property obtained under current-state dependent switching rates.

Abstract

Motivated by networked systems in random environment and controlled hybrid stochastic dynamic systems, this work focuses on modeling and analysis of a class of switching diffusions consisting of continuous and discrete components. Novel features of the models include the discrete component taking values in a countably infinite set, and the switching depending on the value of the continuous component involving past history. In this work, the existence and uniqueness of solutions of the associated stochastic differential equations are obtained. In addition, Markov and Feller properties of a function-valued stochastic process associated with the hybrid diffusion are also proved. In particular, when the switching rates depend only on the current state, strong Feller properties are obtained. These properties will pave a way for future study of control design and optimization of such dynamic systems.

Equations

$$\begin{array}[]{ll}&\displaystyle dX(t)=[A(\alpha(t))X(t)+B(\alpha(t))u(t)]dt+\sigma(\alpha(t))dW(t),\\ &\displaystyle X(s)=x,\hbox{ for }s\leq t\leq T,\end{array}$$

$$\displaystyle J(s,i,x,u(\cdot))=E\Big{[}\!\!\int^{T}_{s}[X^{\top}(t)M(\alpha(t))X(t)+u^{\top}(t)N(\alpha(t))u(t)]dt+X^{\top}(T)DX(T)\Big{]}$$

$$dX(t)=b(X(t),\alpha (t))dt+\sigma (X(t),\alpha (t))dW(t).$$

$$\begin{array}[]{ll}&\displaystyle\mathbb{P}\{\alpha(t+\Delta)=j|\alpha(t)=i,X_{s},\alpha(s),s\leq t\}=q_{ij}(X_{t})\Delta+o(\Delta)\text{ if }i\neq j\ \hbox{ and }\\ &\displaystyle\mathbb{P}\{\alpha(t+\Delta)=i|\alpha(t)=i,X_{s},\alpha(s),s\leq t\}=1-q_{i}(X_{t})\Delta+o(\Delta).\end{array}$$

$$\begin{array}[]{rl}&\!\!\!\displaystyle\Delta_{i1}(\phi)=[0,q_{i1}(\phi)),\ \Delta_{ij}(\phi)=\Big{[}\sum_{k=1,k\neq i}^{j-1}q_{ik}(\phi),\sum_{k=1,k\neq i}^{j}q_{ik}(\phi)\Big{)},j>1,j\neq i.\end{array}$$

$$d\alpha (t)={\int }_{\mathbb{R}}h(X_t,\alpha (t-),z)\mathfrak{p}(dt,dz).$$

$$\{\begin{array}{l}dX(t)=b(X(t),\alpha (t))dt+\sigma (X(t),\alpha (t))dW(t)\\ d\alpha (t)={\int }_{\mathbb{R}}h(X_t,\alpha (t-),z)\mathfrak{p}(dt,dz).\end{array}$$

$$\mathcal{L}f(\varphi ,i)=$$

$$\displaystyle\ +\sum_{j=1,j\neq i}^{\infty}q_{ij}(\phi)\big{[}f(\phi(0),j)-f(\phi(0),i)\big{]}$$

$$=$$

$$\displaystyle\ +\sum_{j=1,j\neq i}^{\infty}q_{ij}(\phi)\big{[}f(\phi(0),j)-f(\phi(0),i)\big{]},$$

$$\begin{array}[]{rl}&\!\!\!\displaystyle f_{k}(x,i)=(\partial/\partial x_{k})f(x,i),\ f_{kl}(x,i)=(\partial^{2}/\partial x_{k}\partial x_{l})f(x,i),\ \hbox{ and }\\ &\!\!\!\displaystyle A(x,i)=(a_{kl}(x,i))_{n\times n}=\sigma(x,i)\sigma^{\top}(x,i),\end{array}$$

$$\underset{t\in [0,T]}{\sup }\{q_{\alpha (t)}(X_t)\}<\infty \text{ a.s.}$$

$$\int_{\mathbb{R}}\big{[}f\big{(}\phi(0),i+h(\phi,i,z)\big{)}-f(\phi(0)),i)\big{]}\mathfrak{m}(dz)=\sum_{j=1,j\neq i}^{\infty}q_{ij}(\phi)\big{[}f(\phi(0),j)-f(\phi(0),i\big{]},$$

$$f(X($$

$$=$$

$$\displaystyle+\int_{0}^{t\wedge\xi_{k}}\int_{0}^{k}\big{[}f\big{(}X(s),\alpha(s-)+h(X_{s},\alpha(s-),z)\big{)}-f(X(s),\alpha(s-))\big{]}\mu(ds,dz)$$

$$=$$

$$\displaystyle+\int_{0}^{t\wedge\xi_{k}}\int_{\mathbb{R}}\big{[}f\big{(}X(s),\alpha(s-)+h(X_{s},\alpha(s-),z)\big{)}-f(X(s),\alpha(s-))\big{]}\mu(ds,dz),$$

$$\mu (ds,dz)=\mathfrak{p}(ds,dz)-\mathfrak{m}(dz)ds.$$

$$f(X(t),\alpha (t))-f(X(0),\alpha (0))={\int }_0^t\mathcal{L}f(X_s,\alpha (s-))ds+M_1(t)+M_2(t)\text{ a.s.,}$$

$$\begin{array}[]{ll}&\!\!\!\displaystyle M_{1}(t)=\int_{0}^{t}\nabla f(X(s),\alpha(s-))\sigma(X(s),\alpha(s-))dW(s),\\ &\!\!\!\displaystyle M_{2}(t)=\int_{0}^{t}\int_{\mathbb{R}}\big{[}f\big{(}X(s),\alpha(s-)+h(X_{s},\alpha(s-),z)\big{)}-f(X(s),\alpha(s-))\big{]}\mu(ds,dz).\end{array}$$

$$\mathbb{E}f(X({\tau }_2),\alpha ({\tau }_2))=\mathbb{E}f(X({\tau }_1),\alpha ({\tau }_1))+\mathbb{E}{\int }_{{\tau }_1}^{{\tau }_2}\mathcal{L}f(X_t,\alpha (t-))dt.$$

$$dX(t)=X(t)\big{[}a(\alpha(t))-b(\alpha(t))X(t)\big{]}dt+\sigma(\alpha(t))X(t)dW(t),$$

$$d\alpha (t)={\int }_{\mathbb{R}}h(X_t,\alpha (t-),z)\mathfrak{p}(dt,dz),$$

$$J(c(\cdot ))=\underset{T\to \infty }{\mathrm{liminf}}\frac{1}{T}\mathbb{E}{\int }_0^T[U(c(t))-D(X(t))]dt,$$

$$dX(t)=[c(t)-\rho (\alpha (t))X(t)]dt+\sigma (X(t),\alpha (t))dW(t).$$

$$|b(x,i)-b(y,i)|+|\sigma (x,i)-\sigma (y,i)|\le L_i|x-y|\forall x,y\in {\mathbb{R}}^n.$$

$$M:=\underset{\varphi \in \mathcal{C},i\in {\mathbb{Z}}_{+}}{\sup }\{q_i(\varphi )\}<\infty .$$

$$dY(t)=b(Y(t),i)dt+\sigma (Y(t),i)dW(t)\text{ for each }i\in {\mathbb{Z}}_{+}.$$

$$d{\tilde{Y}}^{(0)}(t)=b({\tilde{Y}}^{(0)}(t),i_0)dt+\sigma ({\tilde{Y}}^{(0)}(t),i_0)dW(t).$$

$$\begin{array}[]{rl}&\!\!\!\displaystyle\tau_{1}=\inf\{t>0:\int_{0}^{t}\int_{\mathbb{R}}h(\tilde{Y}^{(0)}_{s},i_{0},z)\mathfrak{p}(ds,dz)\neq 0\}\ \hbox{ and }\\ &\!\!\!\displaystyle i_{1}=i_{0}+\int_{0}^{\tau_{1}}\int_{\mathbb{R}}h(\tilde{Y}^{(0)}_{s},i_{0},z)\mathfrak{p}(ds,dz),\end{array}$$

$$d{\tilde{Y}}^{(1)}(t)=b({\tilde{Y}}^{(1)}(t),i_1)dt+\sigma ({\tilde{Y}}^{(1)}(t),i_1)dW(t).$$

$$\begin{array}[]{rl}&\!\!\!\displaystyle\tau_{2}=\inf\{t>\tau_{1}:\int_{\tau_{1}}^{t}\int_{\mathbb{R}}h(\tilde{Y}^{(1)}_{s},i_{1},z)\mathfrak{p}(ds,dz)\neq 0\}\ \hbox{ and }\\ &\!\!\!\displaystyle i_{2}=i_{1}+\int_{\tau_{1}}^{\tau_{2}}\int_{\mathbb{R}}h(\tilde{Y}^{(1)}_{s},i_{1},z)\mathfrak{p}(ds,dz).\end{array}$$

Full text

Keywords. Switching diffusion, past-dependent switching, countable state space, existence and uniqueness of solution, Feller property.

Mathematics Subject Classification. 93E03, 60J60, 60H10, 92D25.

Running Title. Past-Dependent-Switching Diffusion Systems

1 Introduction

Owing to the demand of modeling, analysis, and computation of complex networked systems, much attention has been devoted to building more realistic dynamic system models. It has been well recognized that in many real-world applications, traditional models using continuous processes represented by solutions to deterministic differential equations and stochastic differential equations alone are often inadequate. Arising from control engineering, queueing networks, manufacturing and production planning, parameter estimation, filtering of dynamic systems, ecological and biological systems, and financial engineering, etc., numerous complex systems contain both continuous dynamics and discrete events. The discrete events in these systems are not normally representable by solutions of the usual differential equations. Because of the demand, switching diffusions (also known as hybrid switching diffusions) have drawn growing and resurgent attention. A switching diffusion is a two-component process $(X(t),\alpha(t))$ in which the continuous component $X(t)$ evolves according to the diffusion process whose drift and diffusion coefficients depend on the state of $\alpha(t)$, whereas $\alpha(t)$ takes values in a set consisting of isolated points. Because of their importance, many papers have been devoted to such hybrid dynamic systems; see [11, 21, 27, 28] and the references therein. In their comprehensive treatment of hybrid switching diffusions, Mao and Yuan [15] focused on $\alpha(t)$ being a continuous-time and homogeneous Markov chain independent of the Brownian motion and the generator of the Markov chain being a constant matrix. Realizing the need, treating the two components jointly, Yin and Zhu [26] extended the study to the Markov process $(X(t),\alpha(t))$ by allowing the generator $\alpha(t)$ to depend on the current state $X(t)$. Properties of the underlying process including recurrence, positive recurrence, ergodicity, Feller properties, stability, and invariance among others were investigated. Such study provides us with a clear picture of the underlying processes. Nevertheless, in both of the aforementioned books and most related papers to date, the switching process $\alpha(t)$ is assumed to have a finite state space. One question naturally arises. What happens if the switching process has a countable state space? Much of the argument in [26] relies on the interplay of stochastic processes and the associated systems of partial differential equations. Because the state space of $\alpha(t)$ was assumed to be a finite set, one can essentially treat a system of partial differential equations with a finite number of equations. When we consider problems involving a countable state space, the number of equations becomes infinite. Much more complex situation is encountered. Different methods have to be developed to treat the systems.

There are plenty of real-world applications involving such switching diffusions. Perhaps, one of the most widely used control models in the literature is the so-called LQG (linear quadratic Gaussian regulator) problem; see [3, pp.165-166] for a traditional model. However, for many new applications in networked systems, it has been found that in addition to the random noise represented by Brownian type of disturbances, there is a source of randomness owing to the presence of random environment that can be modeled by a continuous-time Markov chain. Let $\alpha(t)$ be a continuous-time Markov chain with state space ${\mathbb{Z}}_{+}$ (the set of positive integers) and generator $Q$. Consider the controlled dynamic system

[TABLE]

where $X(t)\in{\mathbb{R}}^{n_{1}}$ is the continuous state variable, $u(t)\in{\mathbb{R}}^{n_{2}}$ is the control, $A(i)\in{\mathbb{R}}^{n_{1}\times n_{1}}$ and $B(i)\in{\mathbb{R}}^{n_{1}\times n_{2}}$ are well defined and have finite values for each $i\in{\mathbb{Z}}_{+}$. One may wish to find the optimal control $u(\cdot)$ so that the expected quadratic cost function

[TABLE]

is minimized. The use of $\alpha(t)$ stems from the formulation of discrete events, and the use of ${\mathbb{Z}}_{+}$ enlarges the applicability of previous consideration of finite state space cases. Switched dynamic systems can also be found in, for example, modeling impatient customers and customer abandonment of Markov-modulated service speeds in the heavy-traffic regime and the many-server systems in the Halfin-Whitt regime and the non-degenerate slowdown regime; see [5]. We also refer the reader to Whitt [25] for further reading on limit results in queueing theory and many references therein. In fact, in most of the queueing models, the discrete set is countable rather than finite.

Two more dynamic systems are to be presented in the next section, in which the main interests are to find long-term behavior and control design in an ecological system and to find optimal strategies under long-run average criteria for a pollution management problem. In order to study the aforesaid problems, we first need to ensure that the systems under consideration have unique solutions and that the solutions possess good properties. Motivated by these examples, we take up the challenge of considering a nonlinear hybrid diffusion $(X(t),\alpha(t))$ whose discrete component $\alpha(t)$ has an infinite state space in this paper. Moreover, in lieu of allowing the switching process to depend on the current state $X(t)$ only, we assume that it is past dependent. That is, we assume that the generator of $\alpha(t)$ depends on the past history of the continuous process. This paper provides conditions for the existence and uniqueness of the solutions for given initial data, and to demonstrate the Markov-Feller property of a function-valued stochastic process associated with the equation. Our study will build a bridge for future study on related control systems.

The rest of the paper is organized as follows. The formulation of hybrid switching diffusions with past-dependent switching and countably many possible switching locations is given in Section 2. The existence and uniqueness of solutions to the stochastic equations are then proved under suitable conditions in Section 3. Section 4 studies the Markov and Feller properties of a function-valued stochastic process associated with our equation. The proof for the Feller property is rather complex because the state space of $\alpha(t)$ is infinite, the space of continuous functions is not locally compact, and we do not assume uniform continuity of the switching intensities. In section 5, the strong Feller property of the hybrid diffusion without past-dependent switching is given. Section 6 provides further remarks and points out future research directions. Finally, we provide the proofs of some technical results in an appendix.

2 Formulation

Let $r$ be a fixed positive number. Denote by $\mathcal{C}([a,b],\mathbb{R}^{n})$ the set of $\mathbb{R}^{n}$-valued continuous functions defined on $[a,b]$. In what follows, we mainly work with $\mathcal{C}([-r,0],\mathbb{R}^{n})$, and simply denote it by $\mathcal{C}:=\mathcal{C}([-r,0],\mathbb{R}^{n})$. Denote by $|x|$ the Euclidean norm of $x\in\mathbb{R}^{n}$. For $\phi\in\mathcal{C}$, we use the norm $\|\phi\|=\sup\{|\phi(t)|:t\in[-r,0]\}$. For $y(\cdot)\in\mathcal{C}([-r,\infty),\mathbb{R}^{n})$ and $t\geq 0$, we denote by $y_{t}$ the so-called segment function (or memory segment function) $y_{t}(\cdot):=y(t+\cdot)\in\mathcal{C}$. Let $(\Omega,\mathcal{F},\{\mathcal{F}_{t}\}_{t\geq 0},\mathbb{P})$ be a complete filtered probability space with the filtration $\{\mathcal{F}_{t}\}_{t\geq 0}$ satisfying the usual condition, i.e., it is increasing and right continuous while $\mathcal{F}_{0}$ contains all $\mathbb{P}$-null sets. Let $W(t)$ be an $\mathcal{F}_{t}$-adapted and $\mathbb{R}^{d}$-valued Brownian motion. Suppose $b(\cdot,\cdot):\mathbb{R}^{n}\times{\mathbb{Z}}_{+}\to\mathbb{R}^{n}$ and $\sigma(\cdot,\cdot):\mathbb{R}^{n}\times{\mathbb{Z}}_{+}\to\mathbb{R}^{n\times d}$, where ${\mathbb{Z}}_{+}={\mathbb{N}}\setminus\{0\}=\{1,2,\dots\}$, the set of positive integers. Consider the two-component process $(X(t),\alpha(t))$, where $\alpha(t)$ is a pure jump process taking value in ${\mathbb{Z}}_{+}$, and $X(t)$ satisfies

[TABLE]

We assume that if $\alpha(t-):=\lim_{s\to t^{-}}\alpha(s)=i$, then it can switch to $j$ at $t$ with intensity $q_{ij}(X_{t})$ where $q_{ij}(\cdot):\mathcal{C}\to\mathbb{R}$. When $q_{i}(\phi):=\sum_{j=1,j\neq i}^{\infty}q_{ij}(\phi)$ is uniformly bounded in $(\phi,i)\in\mathcal{C}\times{\mathbb{Z}}_{+}$, and $q_{i}(\cdot)$ and $q_{ij}(\cdot)$ are continuous, one may view the aforementioned assumption as

[TABLE]

However, when $q_{i}(\phi)$ and $q_{ij}(\phi)$ are either discontinuous or unbounded, it does not seem appropriate to use (2.2) to model the switching intensity. To formulate the problem in a general setting without the boundedness and continuity assumptions mentioned above, we construct $\alpha(t)$ as the solution to a stochastic differential equation with respect to a Poisson random measure. We elaborate on the idea below. Let $\mathfrak{p}(dt,dz)$ be a Poisson random measure with intensity $dt\times\mathfrak{m}(dz)$ and $\mathfrak{m}$ is the Lebesgue measure on $\mathbb{R}$ such that $\mathfrak{p}(\cdot,\cdot)$ is independent of the Brownian motion $W(\cdot)$. Let $\tilde{\mathfrak{p}}$ be the Poisson point process associated with $\mathfrak{p}(\cdot,\cdot)$ (see e.g., [23]). Then $\tilde{\mathfrak{p}}$ can lie in a set $A$ with intensity $\mathfrak{m}(A)$, that is, the expected number of Poisson points lying in $A$ during the period $dt$ is $dt\times\mathfrak{m}(A)$. Using this fact, for each $i\in\mathbb{Z}$, we can construct disjoint sets $\{\Delta_{ij}(\phi),j\neq i\}$ such that $\mathfrak{m}(\Delta_{ij}(\phi))=q_{ij}(\phi).$ Let $\tilde{\mathfrak{p}}$ govern the switching of $\alpha(t)$ in the manner that if $\alpha(t-)=i$ and there is a Poisson point in $\Delta_{ij}(X_{t})$ at time $t$, then $\alpha(t)=j$. If $\alpha(t-)=i$ and there is no Poisson point in $\cup_{j\neq i}\Delta_{ij}(X_{t})$ at time $t$, $\alpha(t)$ remains $i$. Using this idea, we formulate the equation for $\alpha(t)$ as follows. For each function $\phi:[-r,0]\to\mathbb{R}^{n}$, and $i\in{\mathbb{Z}}_{+}$, let $\Delta_{ij}(\phi),j\neq i$ be the consecutive left-closed and right-open intervals of the real line, each having length $q_{ij}(\phi)$. That is,

[TABLE]

Define $h:\mathcal{C}\times{\mathbb{Z}}_{+}\times\mathbb{R}\mapsto\mathbb{R}$ by $h(\phi,i,z)=\sum_{j=1,j\neq i}^{\infty}(j-i)\boldsymbol{1}_{\{z\in\Delta_{ij}(\phi)\}}$, where $\boldsymbol{1}_{\{z\in\Delta_{ij}(\phi)\}}=1$ if $z\in\Delta_{ij}$, otherwise $\boldsymbol{1}_{\{z\in\Delta_{ij}(\phi)\}}=0$, is the indicator function. The process $\alpha(t)$ can be defined as a solution to

[TABLE]

The pair $(X(t),\alpha(t))$ is therefore a solution to the system of equations

[TABLE]

A strong solution to (2.3) on $[0,T]$ with initial data $(\phi,i_{0})$ being $\mathcal{C}\times{\mathbb{Z}}_{+}$-valued and $\mathcal{F}_{0}$-measurable random variable, is an $\mathcal{F}_{t}$-adapted process $(X(t),\alpha(t))$ such that

• •

$X(t)$ is continuous and $\alpha(t)$ is cadlag (right continuous with left limits) almost surely (a.s.).

• •

$X(t)=\phi(t)$ for $t\in[-r,0]$ and $\alpha(0)=i_{0}$

• •

$(X(t),\alpha(t))$ satisfies (2.3) for all $t\in[0,T]$ a.s.

We will show in the Appendix that the solution $(X(t),\alpha(t))$ to (2.3), satisfies (2.2) under suitable conditions. Let $f(\cdot,\cdot):\mathbb{R}^{n}\times{\mathbb{Z}}_{+}\mapsto\mathbb{R}$ be twice continuously differentiable in $x$ and bounded in $(x,i)\in\mathbb{R}^{n}\times{\mathbb{Z}}_{+}$. We define the “operator” $\mathcal{L}f(\cdot,\cdot):\mathcal{C}\times{\mathbb{Z}}_{+}\mapsto\mathbb{R}$ by

[TABLE]

where $b(x,i)=(b_{1}(x,i),\dots,b_{n}(x,i))^{\top}$, $\nabla f(x,i)=(f_{1}(x,i),\dots,f_{n}(x,i))\in{\mathbb{R}}^{1\times n}$ and $\nabla^{2}f(x,i)=(f_{ij}(x,i))_{n\times n}$ are the gradient and Hessian of $f(x,i)$ with respect to $x$, respectively, with

[TABLE]

with $z^{\top}$ denoting the transpose of $z$. Suppose that $(X(t),\alpha(t))$ satisfies (2.3) and that for any $T>0$,

[TABLE]

Let $\xi_{k}=\inf\{t>0:q_{\alpha(t)}(X_{t})\geq k\},k\in{\mathbb{Z}}_{+}.$ By noting that $h(X_{t},z)=0$ if $z\notin[0,q_{\alpha(t)}(X_{t}))$ and that

[TABLE]

we have from Itô’s formula (see [2, Theorem 4.4.7]) that

[TABLE]

where $\mu(ds,dz)$ is the compensated Poisson random measure given by

[TABLE]

Under condition (2.5), there exists a random integer $k_{0}=k_{0}(\omega)$ such that $t\wedge\xi_{k}=t$ for any $k>k_{0}$. As a result,

[TABLE]

where $M_{1}(\cdot)$ and $M_{2}(\cdot)$ are local martingales, defined by

[TABLE]

It should be noted that $\mathcal{L}$ is not the generator of the Markov process $(X_{t},\alpha(t))$. However this operator is very useful for analyzing the process $(X(t),\alpha(t))$. In view of (2.7), if $\tau_{1}\leq\tau_{2}$ are stopping times that are bounded above by $T$ a.s., and $f(\cdot,\cdot)$ and $\mathcal{L}f(\cdot,\cdot)$ are bounded and (2.5) holds, then

[TABLE]

Remark 2.1**.**

If $\alpha(t)$ depends on the continuous state, but there is no past dependence (that is, $X_{t}$ is replaced by $X(t)$ in (2.2), and $\phi$ and $\phi(0)$ are replaced by the current state $X(t)=x$ in (2.4), respectively), then $\mathcal{L}$ is indeed the generator of the process $(X(t),\alpha(t))$. Even in this case, the current paper settles the matter of the state space of the switching process being countable thus generalizes the study of finite state space cases as considered in [26].

Example 2.2**.**

This example stems from applications in ecological systems and biological control. Consider the evolution of two interacting species. One is micro, which is described by a logistic differential equation perturbed by a white noise. The other is macro, we assume that its number of individuals follows a birth-death process. Let $X(t)$ be the density of the micro species and $\alpha(t)$ the population of the macro species. The life cycle of a micro species is usually very short, so it is reasonable to assume that the evolution of $X(t)$ can be described by the following past-independent equation

[TABLE]

where $a(i),b(i),\sigma(i)$ are positive constants for each $i\in\mathbb{Z}_{+}$.

On the other hand, the reproduction process of $\alpha(t)$ is assumed to be non-instantaneous. More precisely, suppose the reproduction depends on the period of time from egg formation to hatching, say $r$. Then we have

[TABLE]

where $h(\phi,i,z)=\sum_{j=1,j\neq i}^{\infty}(j-i)\boldsymbol{1}_{\{z\in\Delta_{ij}(\phi)\}}$, $\Delta_{i,i+1}(\phi)=[0,\beta_{i}(\phi))$, $\Delta_{i,i-1}(\phi)=[0,\delta_{i}(\phi))$, $\Delta_{i,j}(\phi)=\emptyset$ if $j\notin\{i-1,i,i+1\}$ or $i=0$. Usually $\beta_{i}(\phi),\delta_{i}(\phi)$ can be given in the integral from $\beta_{i}(\phi)=\int_{-r}^{0}\tilde{\beta}_{i}(t)\phi(t)dt$, $\delta_{i}(\phi)=\int_{-r}^{0}\tilde{\delta}_{i}(t)\phi(t)dt$, for some appropriate weighting functions $\tilde{\beta}_{i},\tilde{\delta}_{i}$. As can be seen from the above, the switching process at $t$ in fact depends on past history of the state $X(\cdot)$. Investigating the interactions between the two species are very important to biological control. A basic biological control problem aims to choose a suitable living organism to control a particular pest (see e.g., [6, 13]). This chosen organism might be a predator, parasite, or disease, which will attack the harmful insect. To design and evaluate effectiveness of a biological control, some questions should be answered first. For example, under which conditions the species will be permanent forever or they will extinct at some instance? Whether or not there is an invariant measure associated with the system under consideration. Mathematically, these questions are related to the stability and ergodicity of the corresponding stochastic systems, which will be studied in a future paper. **

Example 2.3**.**

Pollution management is vitally important and has a significant impact on environment. A major issue is concerned with the tradeoff of pollution accumulation and consumption, which affects environmental policy making. Following the seminal paper of Keeler et al. [10], much work has been devoted to the study of optimal control of dynamic economic systems. In [9], Kawaguchi and Morimoto treated a pollution accumulation problem of maximizing the long-run average welfare using a controlled diffusion model. Assume that an economy consumes some good and meanwhile generates pollution. The pollution stock is gradually degraded and its instantaneous growth rate incorporates a random disturbance with mean zero and constant variance. The social welfare is defined by the utility of the consumption net of the disutility of pollution. The problem is to find optimal consumption strategies for the society in the long-run average sense. Departing from their formulation, we consider an extension of their model. Suppose that there is a switching process $\alpha(t)$ taking values in ${\mathbb{Z}}_{+}$ such that $\alpha(t)$ represents the level of pollution at time $t$. Assume that the stock of pollution at time $t$ is given by $X(t)$, a real-valued process, and there is a positive real-valued function $\rho(\cdot)$ so that for each $i\in{\mathbb{Z}}_{+}$, the rate of pollution decay is $\rho(i)$. The consumption rate (or flow of pollution) is a control process, which is denoted by $c(t)$ at time $t$; the social utility function of the consumption $c$ is denoted by $U(c)$, whereas the social disutility of the pollution stock $x$ is $D(x)$. We say that the consumption rate is admissible if it is ${\cal F}_{t}$-measurable, where ${\cal F}_{t}=\{(X(s),\alpha(s)):s\leq t\}$ such that $0\leq c(t)\leq K_{0}$ for some $K_{0}>0$. The ultimate objective is to maximize the long-run average welfare

[TABLE]

subject to

[TABLE]

Assume that the pollution level $\alpha(\cdot)$ satisfies the conditions (2.2). First, it is reasonable that the level of pollution can be modeled by a continuous-time process taking values in ${\mathbb{Z}}_{+}$. Second, to be more realistic, the pollution level depends on the pollution stock $X(t)$ as well as some past history as given in (2.2). As another generalization of [9], we assume that $\sigma$ in fact depends on $(X(t),\alpha(t))$, and the switching rate depends on some past history of the pollution stock $X(\cdot)$ as in (2.2), and $\sigma^{2}(x,i)>0$ for each $i\in{\mathbb{Z}}_{+}$. Treating the optimal pollution management problem, it is natural to consider the replacement of the average in (2.10) by the average with respect to an invariant measure (if it exists) of the controlled systems. To do so, we need to make sure that (2.11) indeed has an invariant measure. Before this matter can be settled, we need to show that the system has a unique solution for each initial data, and the solution possesses certain desired properties such as Markov and Feller properties.

3 Existence and Uniqueness of Solutions

We are now in a position to prove the existence and uniqueness in the strong sense of a solution with given initial data under suitable conditions. We give several sets of conditions. The main reason is due to the past dependence and the use of ${\mathbb{Z}}_{+}$. First in contrast to the case of switching process staying in a finite set, care needs to be exercised regarding uniformity with respect to the switching set. Second, the past dependence requires careful handling of the use of Lipschitz continuity etc. and the uniformity with respect to the element in the corresponding function spaces. Depending on the preference, Assumptions 3.1 allows certain bounds to be dependent of the switching state $i$, but uniform in the variable in the function space, whereas Assumption 3.2 requires uniformity in the bounds w.r.t. $i$, but requires the past dependent part be localized. Assumptions 3.3 and 3.4 relax the Lipschitz condition to local Lipschitz together with certain growth conditions presented by using bounds with the help of Lyapunov functions.

Assumption 3.1**.**

Assume the following conditions hold.

• (i)

For each $i\in{\mathbb{Z}}_{+}$, there is a positive constant $L_{i}$ such that

[TABLE]

• (ii)

$q_{ij}(\phi)$ is measurable in $\phi\in\mathcal{C}$ for all $i$ and $j\in{\mathbb{Z}}_{+}$. Moreover,

[TABLE]

Theorem 3.1**.**

Under Assumption 3.1, for each initial data $(\xi,i_{0})$, there exists a unique solution $(X(t),\alpha(t))$ to (2.3).

Proof.

It is well-known that part (i) of Assumption 3.1 guarantees the existence and uniqueness of strong solutions to the following diffusion

[TABLE]

Then, given a stopping time $\tau$ and an $\mathcal{F}_{\tau}$-measurable $\mathbb{R}^{n}$-valued random variable $y=y(\tau)$ (depending on $\tau$), there exists a unique strong solution to (3.1) in $[\tau,\infty)$ satisfying $Y(\tau)=y(\tau)$ (see [15, Remark 3.10]). We can now construct the solution to (2.3) with initial data $(\xi,i_{0})$ by the interlacing procedure similar to [2, Chapter 5]. Let $\tilde{Y}^{(0)}(t),t\geq 0$ be the solution with initial data $\xi(0)$ to

[TABLE]

We also set $\tilde{Y}^{(0)}(t)=\xi(t)$ for $t\in[-\tau,0]$. Let

[TABLE]

and $\tilde{Y}^{(1)}(t),t\geq\tau_{1}$ be the solution with $\tilde{Y}^{(1)}_{\tau_{1}}=\tilde{Y}^{(0)}_{\tau_{1}}$ to

[TABLE]

Define

[TABLE]

Note that, in the notation above, $\tilde{Y}^{(k)}_{t}$ is the function $s\in[-r,0]\mapsto\tilde{Y}^{(k)}(t+s).$ Continuing this procedure, let $\tau_{\infty}=\lim\limits_{k\to\infty}\tau_{k}$ and set

[TABLE]

Clearly, $X(t)$ satisfies that for every $t\geq 0$,

[TABLE]

To show that $X(t)$ is a global solution, we need only prove that $\tau_{\infty}=\infty$ a.s. For any $T>0$,

[TABLE]

It follows that $\mathbb{P}\{\tau_{k}\leq T\}\to 0$ as $k\to\infty$. As a result $\tau_{\infty}=\infty$ a.s. By this construction, it can be seen that $X(t)$ is continuous and $\alpha(t)$ is cadlag almost surely. The uniqueness of $(X(t),\alpha(t))$ follows from the uniqueness of $\tilde{Y}^{(k)}(t)$ on $[\tau_{k},\tau_{k+1}]$ and the uniqueness of $i_{k}$ defined by

[TABLE]

This concludes the proof. ∎

Assumption 3.2**.**

Assume the following conditions hold.

• (i)

There is a positive constant $L$ such that

[TABLE]

• (ii)

$q_{ij}(\phi)$ is measurable in $\phi\in\mathcal{C}$ for each $(i,j)\in{\mathbb{Z}}_{+}^{2}$. Moreover, for any $H>0$,

[TABLE]

Remark 3.2**.**

We can use either Assumption 3.1 or Assumption 3.2 to obtain the existence and uniqueness of solutions to (2.3). Recall that now ${\mathbb{Z}}_{+}$ is a countable set, so care must be taken to distinct it with a finite state case. In Assumption 3.1, the Lipschitz constants of $b(\cdot,i),\sigma(\cdot,i)$ depend on $i$, and $q_{i}(\phi)$ is assumed to be bounded uniformly in $(\phi,i)\in\mathcal{C}\times\mathbb{Z}_{+}$. In contrast, the uniform boundedness of $q_{i}(\phi)$ is relaxed, but the Lipschitz constant of $b(\cdot,i),\sigma(\cdot,i)$ is assumed to be in $i\in{\mathbb{Z}}_{+}$.

Theorem 3.3**.**

Under Assumption 3.2, for each initial data $(\xi,i_{0})$, there exists a unique solution $(X(t),\alpha(t))$ to (2.3).

Proof.

Without loss of generality, we may assume that $(\xi,i_{0})$ is bounded, since we can use the truncation method in [4, Theorem 3 in §6] to obtain the result for general $(\xi,i_{0})$ once we have proved for the case $(\xi,i_{0})$ being bounded. Construct the process $(X(t),\alpha(t))$ as in the proof of Theorem 3.1. We need to show that $\tau_{\infty}=\infty$ a.s. Following the proof of [14, Lemma 3.2, p. 51], there is a $K=K(T)$ such that

[TABLE]

As a result, for any $\varepsilon>0$, there is an $H_{\varepsilon}$ such that

[TABLE]

Let $\eta_{H_{\varepsilon}}=\inf\{t\geq 0:\|X_{t}\|\geq H_{\varepsilon}\}$ and $M_{H_{\varepsilon}}=\sup_{\phi\in\mathcal{C},\|\phi\|\leq H_{\varepsilon},i\in{\mathbb{Z}}_{+}}\{q_{i}(\phi)\}<\infty.$ Then

[TABLE]

For sufficiently large $k$, we have

[TABLE]

From (3.6) and (3.8), $\mathbb{P}\{\tau_{k}\geq T\}\geq\mathbb{P}(\{\tau_{k}\wedge T<\eta_{H_{\varepsilon}}\}\cap\{\tau_{k}>T\wedge\eta_{H_{\varepsilon}}\})\geq 1-\varepsilon$ for sufficiently large $k$. Thus, we obtain that $\mathbb{P}\{\tau_{\infty}\geq T\}\geq 1-\varepsilon$. It holds for every $T>0$ and $\varepsilon>0$, so we obtain the desired result. ∎

Remark 3.4**.**

To obtain the existence and uniqueness of solutions, Assumptions 3.1 and 3.2 can be relaxed by replacing the global Lipschitz conditions with local Lipschitz conditions together with Lyapunov-type functions. To be specific, let $V(\cdot):\mathbb{R}^{n}\mapsto\mathbb{R}$ be twice continuously differentiable in $x$. For each $i\in{\mathbb{Z}}_{+}$, let \mathcal{L}_{i}V(x)=\nabla V(x)b(x,i)+\dfrac{1}{2}\operatorname{tr}\Big{(}\nabla^{2}V(x)A(x,i)\Big{)}. For instance (1) of Assumption 3.1 and (1) of Assumption 3.2 can be replaced by the following Assumptions 3.3 and 3.4, respectively.

Assumption 3.3**.**

Assume the following conditions hold.

• (i)

For each $H>0$, $i\in{\mathbb{Z}}_{+}$, there is a positive constant $L_{H,i}$ such that

[TABLE]

• (ii)

For each $i\in{\mathbb{Z}}_{+}$, there exist a twice continuously differentiable function $V_{i}(x)$ and a constant $C_{i}>0$ such that

[TABLE]

Assumption 3.4**.**

Assume the following conditions hold.

• (i)

For each $H>0$, $i\in{\mathbb{Z}}_{+}$, there is a positive constant $L_{H,i}$ such that

[TABLE]

• (ii)

There exist a twice continuously differentiable function $V(x)$ and a constant $C>0$ independent of $i\in{\mathbb{Z}}_{+}$ such that

[TABLE]

Theorem 3.5**.**

For given initial data $(\xi,i_{0})$, there exists a unique solution $(X(t),\alpha(t))$ to (2.3) if either of the following conditions is satisfied

• •

Assumption 3.3 and (ii) of Assumption 3.1,

• •

Assumption 3.4 and (ii) of Assumption 3.2.

Proof.

It is well known that Assumption 3.3 guarantees the existence and uniqueness of solutions to (3.1). Hence, if (ii) in Assumption 3.1 is satisfied, we can prove the desired result by using the proof of Theorem 3.1. Now, suppose Assumption 3.4 and (ii) of Assumption 3.2 hold. Similar to the proof of Theorem 3.3, we can assume that $(\xi,i_{0})$ is bounded. Consider $X(t)$ and define $\tau_{k}$ as in the proof of Theorem 3.1. Then $X(t)$ is the solution with initial data $(\xi,i_{0})$ to (2.3) on $[0,T\wedge\tau_{k})$ for any $T>0,k\in{\mathbb{Z}}_{+}$. We have from the generalized Itô formula that

[TABLE]

where $\eta_{H}=\inf\{t\geq 0:|X(t)|>H\}$. Using the estimate above and the argument in [15, Theorem 3.19], we can show that

[TABLE]

In view of the property \lim\limits_{R\to\infty}\Big{(}\inf\{V(x):|x|\geq R\}\Big{)}=\infty, for any $\varepsilon>0$, there is $H_{\varepsilon}>0$ such that

[TABLE]

Then, proceeding similarly as in the proof of Theorem 3.3 yields the existence and uniqueness of solutions with initial data $(\xi,i_{0})$ to (2.3). ∎

Example 3.6**.**

(cont. of Example 2.2) We come back to Example 2.2 in Section 1. We want to show that $X(t)>0$ for all $t\geq 0$ under certain conditions. To proceed, we can set $Y(t)=\ln X(t)$ to obtain

[TABLE]

To demonstrate (2.8) and (2.9) has a unique solution with $X(t)>0$ for all $t\geq 0$, it is equivalent to show that (3.9) and (2.9) has a strong solution on $[0,\infty)$. Let $V(y)=e^{y}+e^{-y}$. By direct calculation,

[TABLE]

where $c(i)=\max\limits_{y\in\mathbb{R}}\{b(i)+(2a(i)-\sigma^{2}(i))e^{y}-b(i)e^{2y}\}$. Applying Theorem 3.3, we can see that the equation has a unique solution if one of the following is satisfied

• •

$\beta_{i}(\phi)+\delta_{i}(\phi)$ is bounded uniformly in $\phi\in\mathcal{C}_{+}:=\{\psi\in\mathcal{C}:\psi(t)>0\,\forall t\in[-r,0]\}$ and $i\in\mathbb{Z}_{+}$.

• •

$c(i)$ and $\sigma^{2}(i)-a(i)$ are bounded above uniformly and for each $i\in\mathbb{Z}_{+}$, $\beta_{i}(\phi)+\delta_{i}(\phi)$ is bounded in each compact subset of $\phi\in\mathcal{C}_{+}$.

It can be shown by applying the result of the next section that the process $(Y_{t},\alpha(t))$ has the Markov-Feller property if $\beta_{i}(\cdot)$ and $\delta_{i}(\cdot)$ are continuous in addition to one of the above conditions.

Example 3.7**.**

(cont. of Example 2.3) To study the long-run average control problem in Example 2.3, it is important to make sure that the system under consideration processes ergodicity. Before the ergodicity can be verified, we need (2.11) has a unique solution for each initial condition. Denote the control set by $\widetilde{K}$ and assume it is a compact and convex set. Using a relaxed control representation $m_{t}(\cdot)$ (see [12]) to represent the consumption rate $c(\cdot)$, we can rewrite (2.11) as

[TABLE]

Assume that for each $i\in{\mathbb{Z}}_{+}$, $\sigma(x,i)$ satisfies the conditions in Assumption 3.1 (i), and $Q(\phi)$ satisfies Assumption 3.1 (ii). Then the conditions of Theorem 3.1 are all verified. As a result, (3.10) has a unique solution for each initial condition.

4 Markov and Feller Properties

This section establishes the Markov and Feller properties of the process $(X_{t},\alpha(t))$. While the Markov property can be derived by the well-known arguments, it requires much more efforts to obtain the Feller property. As already seen in the previous section, the past dependence and the use of ${\mathbb{Z}}_{+}$ make the analysis more complex than that of the switching diffusions with diffusion-dependent switching living in a finite set. To overcome the difficulties, in this section, we carry out the analysis by introducing some auxiliary or intermediate processes. First, it would be better if we could untangle the past dependence of the switching process and the infinity of the cardinality of its state space. For this purpose, we introduce a continuous-time Markov chain independent of the past and continuous state; we call this process $\gamma(t)$. Then naturally, associated with $\gamma(t)$, we examine a pair of process $(Z(t),\gamma(t))$. Even after this introduction, in the analysis, we still need to look into the details of the switching process $\alpha(t)$ such as when it jumps and the post jump location etc. To do so, we introduce another auxiliary process $Y(t)$, which is a “fixed”-$i$ process. We then have another pair of processes $(Y(t),\beta(t))$ to deal with. These auxiliary processes help us to establish the desired results. Their connections and interactions will be further specified in what follows.

First, note that the Brownian motion and the Poisson point process associated to $\mathfrak{p}(dt,dz)$ possess stationary strong Markov property, that is, for any finite stopping time $\eta$, $\{W^{*}(t)\}_{t\geq 0}=\{W(t+\eta)-W(\eta)\}_{t\geq 0}$ is an $\mathcal{F}^{*}_{t}$-Brownian motion and $\mathfrak{p}^{*}([t,t+s)\times U)=\mathfrak{p}([t+\eta,t+s+\eta)\times U)$ is a Poisson random measure with density $dt\times\mathfrak{m}(dz)$ (see [23, Theorem 101]). Hence, by standard arguments, we can obtain the following theorem whose proof is omitted. In fact, the theorem can be proved essentially by imitating the proof in [18, Chap. 5], [2, Chap. 6], or [16, Chap. 7].

Theorem 4.1**.**

Assume that the hypotheses of Theorem 3.1, or Theorem 3.3, or Theorem 3.5 are satisfied. Let $(X(t),\alpha(t))$ be a solution to (2.3). Then $(X_{t},\alpha(t))$ is a homogeneous strong Markov process taking value in $\mathcal{C}\times{\mathbb{Z}}_{+}$ with transition probabilities

[TABLE]

where $X^{\phi,i}(t)$ is the solution to (2.3) with initial data $(\phi,i)\in\mathcal{C}\times{\mathbb{Z}}_{+}$.

We proceed with obtaining the Feller property of $(X_{t},\alpha(t))$. Assuming that the hypotheses of Theorem 3.1, or Theorem 3.3, or Theorem 3.5 are satisfied leads to the existence and uniqueness of strong solutions. Next, we introduce an auxiliary hybrid diffusion with Markov switching. Let $\gamma^{i}(t)$ be a Markov chain starting at $i$ with generator $\tilde{Q}=(\rho_{ij})$ for $(i,j)\in{{\mathbb{Z}}_{+}\times{\mathbb{Z}}_{+}}$, where $\rho_{ii}=-1$ and $\rho_{ij}=2^{-j}$ if $j<i$ and $\rho_{ij}=2^{-j+1}$ if $j>i$, that is,

[TABLE]

We recursively define a sequence of stopping times $\{\theta^{i}_{k}\}$ with $\theta^{i}_{k}$ being the first jump time of $\gamma^{i}(t)$ after $\theta^{i}_{k-1}$ as follows

[TABLE]

For $(\phi,i)\in\mathcal{C}\times{\mathbb{Z}}_{+}$, let $Z^{\phi,i}(t)$ be the solution to

[TABLE]

satisfying $Z^{\phi,i}(t)=\phi(t)$ in $[-r,0]$ and $\gamma(0)=i$.

Similar to Girsanov’s theorem, which tells us how to convert an Itô process to a Brownian motion under a change of measure, we aim to establish a change of measure allowing us to “convert” a hybrid diffusion with past-dependent switching to a hybrid diffusion with Markov switching. To establish such a change of measure, we need to find the distribution of jump times of $\alpha(t)$. Because of the interactions between $\alpha(t)$ and $X(t)$, we need to introduce another auxiliary (or intermediate) process, which helps to examine the distribution of the jump times of $\alpha(t)$. Let $(Y^{\phi,i}(t),\beta^{\phi,i}(t))$ be the solution to

[TABLE]

satisfying $Y^{\phi,i}(t)=\phi(t)$ in $[-r,0]$ and $\beta^{\phi,i}(0)=i$.

By the definition, $\alpha^{\phi,i}(t)=\beta^{\phi,i}(t),X^{\phi,i}(t)=Y^{\phi,i}(t)$ up to the first jump time of the two process $\alpha(t)$ and $\beta(t)$. There is an advantage working with $(Y^{\phi,i}(t),\beta^{\phi,i}(t))$. Unlike the pair $(X(t),\alpha(t))$ in which $\alpha(t)$ depends on the continuous state, the process $Y^{\phi,i}(t)$ evolving for a fixed discrete state $i$ that does not depend on $\beta^{\phi,i}(t)$. Thus, it is easier to examine, for example, the first jump time of $\beta^{\phi,i}(t)$ (or $\alpha^{\phi,i}(t)$).

Next we recursively define sequences of stopping times associated with $\beta(t)$ and $\alpha(t)$ so that $\lambda^{\phi,i}_{k}$ and $\tau^{\phi,i}_{k}$ are the first jump times of the processes $\beta^{\phi,i}(t)$ and $\alpha^{\phi,i}(t)$ after $\lambda^{\phi,i}_{k-1}$ and $\tau^{\phi,i}_{k-1}$, respectively. More specifically, for $k\in{\mathbb{Z}}_{+}$, let

[TABLE]

and

[TABLE]

To simplify the notation, we put

[TABLE]

and

[TABLE]

where we use the subscript $k$ with parentheses to avoid confusion with the function-valued processes $X^{\phi,i}_{t},Y^{\phi,i}_{t},Z^{\phi,i}_{t}$ at $t=k$.

Lemma 4.2**.**

Let $g:\mathcal{C}\times\mathbb{R}_{+}\times{\mathbb{Z}}_{+}\mapsto\mathbb{R}$ be a bounded and measurable function, and $\mathcal{F}^{W}_{T}$ be the $\sigma$-algebra generated by $\{W(t),t\in[0,T]\}$. The following assertions hold:

1. (i)

\mathbb{P}\big{(}\{\lambda_{1}^{\phi,i}>t\}\big{|}\mathcal{F}^{W}_{T}\big{)}=\mathbb{E}\Big{[}\boldsymbol{1}_{\{\lambda_{1}^{\phi,i}>t\}}\Big{|}\mathcal{F}^{W}_{T}\Big{]}=\exp\big{(}-\displaystyle\int_{0}^{t}q_{i}(Y^{\phi,i}_{s})ds\big{)}\,\ \forall\,t\in[0,T].** 2. (ii)

\mathbb{E}\Big{[}g(Y^{\phi,i}_{(1)},\lambda_{1}^{\phi,i},\beta^{\phi,i}_{1})\boldsymbol{1}_{\{\lambda_{1}^{\phi,i}\leq T\}}\Big{|}\mathcal{F}^{W}_{T}\Big{]}=\displaystyle\sum_{j=1,j\neq i}^{\infty}\int_{0}^{T}g(Y_{t},t,j)q_{ij}(Y_{t})\exp(-\int_{0}^{t}q_{i}(Y_{s})ds)dt.**

As indicated previously, it is difficult to estimate the difference of $X^{\phi_{1},i}_{t}$ and $X^{\phi_{2},i}_{t}$ because the states of $\alpha^{\phi_{1},i}(t)$ and $\alpha^{\phi_{2},i}(t)$  may differ significantly due to the continuous state dependence. In contrast, it is considerably easier to compare $Z^{\phi_{1},i}_{t}$ and $Z^{\phi_{2},i}_{t}$ because of the continuous-state-dependent switching is replaced by the continuous-state-independent Markov chain. With help of the intermediate process $(Y(t),\beta(t))$ and Lemma 4.2, we obtain the following change of measure formula, which is a bridge to connect the continuous-state-dependent and continuous-state-independent processes.

Proposition 4.3**.**

For any $T>0$, let $f(\cdot,\cdot):\mathcal{C}\times{\mathbb{Z}}_{+}\mapsto\mathbb{R}$ be a bounded continuous function. For any $l=0,1,\dots$, any $i_{k}\in{\mathbb{Z}}_{+}$ with $i_{k}\neq i_{k+1}$ and $k=1,\dots,l+1$, and any $(\phi,i)\in\mathcal{C}\times{\mathbb{Z}}_{+}$,

[TABLE]

Remark 4.4**.**

The proofs of Lemma 4.2 and Proposition 4.3 will be given in the Appendix. We are now in a position to prove the Feller property for the solution to (2.3). In addition to the sufficient conditions for the existence and uniqueness of solution, we prove the Feller property of the solution only with an additional condition that $q_{ij}(\phi)$ is continuous in $\phi$ for any $i,j\in{\mathbb{Z}}_{+}$. There are some difficulties because the process $\{X_{t}\}$ takes value in an infinite dimensional Banach space and the switching $\{\alpha(t)\}$ has an infinite state space. Moreover, although we suppose that $q_{ij}(\phi)$ is continuous, neither the uniform continuity in $\phi\in\mathcal{C}$ nor equi-continuity in $i,j\in{\mathbb{Z}}_{+}$ is assumed. Because of these difficulties, we divide the proof into several steps. First, we make the following assumptions, which will be relaxed later.

Assumption 4.1**.**

Assume the following conditions hold.

1. (i)

For each $i\in{\mathbb{Z}}_{+}$, $b(x,i)$ and $\sigma(x,i)$ are Lipschitz continuous functions that are vanishing outside $\{x:|x|\leq R\}$ for some $R>0$. 2. (ii)

$M:=\sup\{q_{i}(\phi):i\in{\mathbb{Z}}_{+},\phi\in\mathcal{C}\}<\infty$. 3. (iii)

For each $i,j\in{\mathbb{Z}}_{+},j\neq i$, $q_{i}(\cdot)$ and $q_{ij}(\cdot)$ are continuous on $\mathcal{C}$.

Before applying (4.2) to prove the continuous dependence of $u_{f}(\phi,i)=\mathbb{E}_{\phi,i}f(X_{T},\alpha(T))$ on $(\phi,i)$, we first need the following lemma.

Lemma 4.5**.**

Assume that Assumption 4.1 is satisfied. Let $(\phi_{0},i_{0})\in\mathcal{C}\times{\mathbb{Z}}_{+}$ with $\|\phi_{0}\|\leq R$ and $T>0$. For each $\Delta>0$, there exist $m=m(\Delta)\in{\mathbb{Z}}_{+}$, $n_{m}=n_{m}(\Delta)\in{\mathbb{Z}}_{+}$, and $d_{m}=d_{m}(\Delta)>0$ such that

[TABLE]

where $N_{k}=\{1,\dots,k\}$.

This lemma allows us to confine our attention to a finite subset of ${\mathbb{Z}}_{+}$ (the state space of $\alpha^{\phi,i_{0}}(\cdot)$) and a finite number of jumps when $\phi$ is close to $\phi_{0}$. It is a crucial step in providing some uniform estimates because we do not assume the equi-continuity of $q_{ij}(\cdot)$ in either $i$ or $j$. Since the switching intensity of $\alpha^{\phi,i_{0}}(t)$ depends on $X_{t}^{\phi,i_{0}}$, in order to obtain Lemma 4.5, we need to show that with an arbitrarily large probability, $X_{t}^{\phi,i_{0}},t\in[0,T]$ belongs to a compact set in $\mathcal{C}$ for any $\phi$ sufficiently close to $\phi_{0}$. Note that under some suitable conditions, sample paths of a diffusion process in a finite interval $[0,T]$ are Hölder continuous. Thus, it is easy to find a compact set in which sample paths of a diffusion process lie with a large probability. Our arguments rely on this fact. However, the initial data $\phi$ of our process $X(t)$ does not always satisfy the Hölder continuity. Moreover, $X(t)$ depends on the state of $\alpha(t)$. We therefore need to introduce the following operator, which is motivated by merging trajectories of $X(t)$ at jump times. For $\mathcal{A},\mathcal{B}\subset\mathcal{C}$, we define the set of continuous functions that are formed by merging functions in $\mathcal{A}$ and $\mathcal{B}$ as follows.

[TABLE]

By virtue of the Arzelá-Ascoli theorem, if $\mathcal{A}$ and $\mathcal{B}$ are compact, so is $\mathcal{A}\uplus\mathcal{B}$. Using this fact and the Hölder continuity of sample paths of a diffusion process, we can find a suitable compact set to which $X_{t}^{\phi,i_{0}},t\in[0,T],$ belongs with a large probability for any $\phi$ which is sufficiently close to $\phi_{0}$. Then, Lemma 4.5 can be proved. The details of the proof are postponed to the Appendix. Now, we point out some nice properties of the diffusion process with Markov switching $(Z(t),\gamma(t))$, which are useful to compare the sample paths of $Z(t)$ with different initial values.

Lemma 4.6**.**

Fix $i_{0}\in{\mathbb{Z}}_{+}$. For each $k\in{\mathbb{Z}}_{+}$ and $\varepsilon>0$, there is an $\hbar_{k}=\hbar_{k}(\varepsilon)>0$ such that

[TABLE]

and

[TABLE]

where $\iota_{k}=\inf\{t>0:\gamma^{i_{0}}(t)>k\}$ and $\overline{C}$ is some positive constant.

Proof.

Since $b(x,i)$ and $\sigma(x,i)$ are Lipschitzian in $x$ uniformly in $N_{k}$, by standard arguments (e.g., [15, Theorem 3.23]), we can show that

[TABLE]

Using the Kolmogorov-Centsov theorem, we obtain the first inequality. The details are similar to the proof of Lemma 4.5 in the Appendix. The second claim is proved in the same manner as that of [26, Lemma 2.14]. ∎

Having Lemmas 4.5 and 4.6, we are ready to use the change of measure (4.2) to prove the Feller property of $(X_{t},\alpha(t))$ under Assumption 4.1.

Proposition 4.7**.**

Suppose that Assumption 4.1 is satisfied. Let $f(\cdot,\cdot):\mathcal{C}\times{\mathbb{Z}}_{+}\mapsto\mathbb{R}$ be continuous and bounded. Then for any $T>0$, $u_{f}(\phi,i)=\mathbb{E}f(X_{T}^{\phi,i},\alpha^{\phi,i}(T))$ is a continuous function in $\phi\in\mathcal{C}$.

Proof.

We suppose without loss of generality that $|f(\phi,i)|\leq 1\,\forall\,(\phi,i)\in\mathcal{C}\times{\mathbb{Z}}_{+}$. Fix $(\phi_{0},i_{0})\in\mathcal{C}\times{\mathbb{Z}}_{+}$. We show that for any $\Delta>0$, there exists $d^{*}=d^{*}(\Delta,\phi_{0},i_{0})>0$ such that

[TABLE]

In view of Lemma 4.5, there are $m$, $n_{m}\in{\mathbb{Z}}_{+}$, and $d_{m}>0$ such that

[TABLE]

Let $\varepsilon=\varepsilon(\Delta)>0$ (to be specified later). Let $\hbar_{k}$ be as in Lemma 4.6. Denote

[TABLE]

and $\tilde{\mathcal{K}}=\{\phi_{0}\}\uplus\tilde{\mathcal{H}}$. By the compactness of $\tilde{K}$, there is a $\tilde{d}_{m}>0$ such that

[TABLE]

if $\phi\in\tilde{\mathcal{K}},i,j\in N_{n_{m}}$ and $\|\psi-\phi\|<\tilde{d}_{m}$. In view of Lemma 4.6, we can choose $\widehat{d}_{m}>0$ such that

[TABLE]

Let $A^{\phi}$ be the event $\{\tau^{\phi,i_{0}}_{m}\leq T<\tau^{\phi,i_{0}}_{m+1},\iota_{n_{m}}>T\}$ and $l(T)$ be the number of jumps up to time $T$ of $\gamma^{i_{0}}(t)$. It follows from Proposition 4.3 that

[TABLE]

where

[TABLE]

Let $D_{m}^{\phi}$ be the event

[TABLE]

Using (4.7) and the estimates in [26, Lemma 2.17], we obtain for $l\geq 1$,

[TABLE]

where $K$ is a constant depending only on $T,m,n_{m}.$

Note that if $\omega\in\{\theta_{l}^{i_{0}}\leq T<\theta_{l+1}^{i_{0}},\iota_{n_{m}}>T\}\cap D_{m}^{\phi}$, then $Z^{\phi_{0},i_{0}}_{t}\in\tilde{\mathcal{K}}$ and $\|Z^{\phi,i_{0}}_{t}-Z^{\phi_{0},i_{0}}_{t}\|\leq\tilde{d}_{m}\,\forall t\in[0,T]$ which implies in view of (4.5) that

[TABLE]

On the other hand, Lemma 4.6 and (4.6) imply that

[TABLE]

Hence for $\|\phi-\phi_{0}\|\leq\widehat{d}_{m}$, we have that

[TABLE]

Note that

[TABLE]

which implies

[TABLE]

Choosing $\varepsilon=\dfrac{\Delta}{2K(4+3M)}$, we have from (4.8) and (4.9) that

[TABLE]

if $\|\phi-\phi_{0}\|<d^{*}:=\dfrac{d_{m}}{2}\wedge\widehat{d}_{m}$. ∎

With the above technical preparations, we are now in a position to prove the main theorem of this section. By using truncation arguments, we can obtain the Feller property of $(X_{t},\alpha(t))$ even if $b(\cdot,i),\sigma(\cdot,i)$ do not vanish outside a bounded region and $q_{i}(\phi)$ is not bounded. The precise condition is given below.

Theorem 4.8**.**

Let either Assumption 3.1 or Assumption 3.2 be satisfied. Assume further that $q_{ij}(\cdot)$ is a continuous function for any $i,j\in{\mathbb{Z}}_{+}$. Then the solution to (2.3) has the Feller property.

Proof.

Let $f(\cdot,\cdot):\mathcal{C}\times{\mathbb{Z}}_{+}\mapsto\mathbb{R}$ be a continuous function with $|f(\phi,i)|\leq 1\,\forall\,(\phi,i)\in\mathcal{C}\times{\mathbb{Z}}_{+}$. Fix $R>0,T>0$. Suppose that $\|\phi_{0}\|<R$. Under the hypotheses of Theorem 3.1, or Theorem 3.3, or Theorem 3.5, it is shown in the proofs of those theorems that for any $\varepsilon>0$, there exists an $\tilde{R}>0$ such that

[TABLE]

Let $\Phi(x):\mathbb{R}^{n}\mapsto\mathbb{R}$ be a twice continuously differentiable satisfying $\Phi(x)=1$ if $|x|\leq\tilde{R}$ and $\Phi(x)=0$ if $|x|\geq\tilde{R}+1$. Let $(\tilde{X}^{\phi,i_{0}}_{t},\tilde{\alpha}^{\phi,i_{0}}(t))$ be the solution with initial data $(\phi,i_{0})$ to

[TABLE]

Then $(\tilde{X}^{\phi,i_{0}}(t),\tilde{\alpha}^{\phi,i_{0}}(t))=(X^{\phi,i_{0}}(t),\alpha^{\phi,i_{0}}(t))$ up to the time that $\|X^{\phi,i_{0}}_{t}\|>\tilde{R}$, which combined with (4.10) implies

[TABLE]

where $\tilde{\Omega}_{\phi,i_{0}}:=\{\tilde{X}^{\phi,i_{0}}_{T}=X^{\phi,i_{0}}_{T},\tilde{\alpha}^{\phi,i_{0}}(T)=\alpha^{\phi,i_{0}}(T)\}$. As a result, if $\|\phi\|<R$, we have

[TABLE]

It follows from Proposition 4.7 that there exists a $\delta\in(0,1)$ such that if $\|\phi-\phi_{0}\|<\delta$, we have

[TABLE]

Since $\|f\|\leq 1$, we can easily obtain from (4.12) and (4.13) that

[TABLE]

The proof of the theorem is complete. ∎

5 Feller Property of Hybrid Diffusion without Past Dependence

Now, we suppose that the $q_{ij},i,j\in{\mathbb{Z}}_{+}$ associated with $\alpha(t)$ depend only the current state of $X(t)$. To be more precise $q_{ij}(\cdot)$ is a function from $\mathbb{R}^{n}$ to $\mathbb{R}$ for each $(i,j)\in{\mathbb{Z}}_{+}\times{\mathbb{Z}}_{+}$. As a special case of the hybrid diffusion with past-dependent switching, we obtain the following theorem.

Theorem 5.1**.**

Assume that $q_{ij}(\cdot)$ is a continuous function for any $i,j\in{\mathbb{Z}}_{+}$. Assume further that one of the following conditions is satisfied:

1. (A)

Assumption 3.3 and $q_{i}(y)=\sum_{j\neq i}q_{ij}(y)$ is bounded uniformly in $(i,y)\in{\mathbb{Z}}_{+}\times\mathbb{R}^{n}$. 2. (B)

Assumption 3.4 and $q_{i}(y)$ is bounded uniformly in $(i,y)\in{\mathbb{Z}}_{+}\times K$ for each compact subset $K$ of $\mathbb{R}^{n}$.

Then the unique solution to (2.3) is a Markov process having the Feller property.

Remark 5.2**.**

If for each discrete state $i\in{\mathbb{Z}}_{+}$, the diffusion $Y^{(i)}(t)$, which is the solution process to

[TABLE]

has the strong Feller property, we do not need the continuity of $q_{ij}(\cdot)$ to get the Feller property of $(X(t),\alpha(t))$. In fact, we will obtain a stronger result, namely, the strong Feller property. The condition for the strong Feller property of $Y^{(i)}(t)$ is essentially the ellipticity of $A(x,i)$ or the Hörmander condition for hyperellipticity (see e.g., [17, 24]).

Theorem 5.3**.**

Assume that $q_{ij}(\cdot)$ is measurable for any $i,j\in{\mathbb{Z}}_{+}$ and either (A) or (B) in Theorem 5.1 holds. If for each $i\in{\mathbb{Z}}_{+}$, the solution process $Y^{(i)}(t)$ to (5.1) has the strong Feller property, then the unique solution to (2.3) has the strong Feller property, that is, for each bounded measurable function $g(y,i):\mathbb{R}^{n}\times{\mathbb{Z}}_{+}\to\mathbb{R}$, the function $(x,i)\to\mathbb{E}g(X^{x,i}(T),\alpha^{x,i}(T))$ is continuous for each $T>0$.

Proof.

We assume without loss of generality that $|g(z,i)|\leq 1\,\forall z\in\mathbb{R}^{n},i\in{\mathbb{Z}}_{+}$. Let $Y^{y,i}(t)$ be the solution with initial data $y$ to (5.1). Fix $(x,i)\in\mathbb{R}^{n}\times{\mathbb{Z}}_{+}$ and $\varepsilon>0$. Under the assumption $(A)$ or $(B)$, we can show that for each $x\in\mathbb{R}^{n}$, there is a $K>0$ satisfying

[TABLE]

where $\Omega_{\varepsilon}^{y,i}=\{|Y^{y,i}(t)|\leq K\,\forall t\in[0,1]\}$, $(\Omega_{\varepsilon}^{y,i})^{c}=\Omega\setminus\Omega_{\varepsilon}^{y,i}$. Let $M=\sup_{i\in{\mathbb{Z}}_{+},|z|\leq K}q_{i}(z)$ and $t_{0}\in(0,1)$ satisfying $1-\exp\{-Mt_{0}\}<\dfrac{\varepsilon}{16}$. It follows from (5.2) and (i) of Lemma 4.2 that

[TABLE]

where

[TABLE]

Denote \Phi(y,i):=\mathbb{E}g\big{(}X^{y,i}(T-t_{0}),\alpha^{y,i}(T-t_{0})\big{)}. The condition $|g(y,i)|\leq 1$ implies $|\Phi(y,i)|\leq 1$ for all $y\in\mathbb{R}^{n},i\in{\mathbb{Z}}_{+}$. By the strong Feller property of $Y^{(i)}(t)$, there is a $\delta>0$ such that

[TABLE]

By the strong Markov property of $X(t)$, we have

[TABLE]

Applying (i) of Lemma 4.2, we obtain

[TABLE]

Note that if $|g(z,i)|\leq 1\,\forall z\in\mathbb{R}^{n},i\in{\mathbb{Z}}_{+}$ then $|\Phi(z,i)|\leq 1\,\forall z\in\mathbb{R}^{n},i\in{\mathbb{Z}}_{+}$. We have the following estimates for $y\in B(x,\delta)$ using (5.2), (5.3), (5.4), and the fact that $|g(z,i)|,|\Phi(z,i)|\leq 1\,\forall z\in\mathbb{R}^{n},i\in{\mathbb{Z}}_{+}$.

[TABLE]

[TABLE]

[TABLE]

Applying estimates (5.4), (5.7), (5.8), and (5.9) to (5.5) and (5.6), we have

[TABLE]

The proof is complete. ∎

Remark 5.4**.**

Sufficient conditions for the strong Feller property of $(X(t),\alpha(t))$, in which the rates of switching $q_{ij}$ for $i,j\in{\mathbb{Z}}_{+}$ depend only on the current continuous state $X(t)$, was obtained in Shao [20]. However, the conditions there are restrictive. To obtain the strong Feller property, it is assumed in [20] that $q_{ij}(x),b(x,i)$ and $\sigma(x,i)$ are Lipschitz in $x$ uniformly in $i\in{\mathbb{Z}}_{+}$. The ellipticity of $A(x,i)$ is also assumed to be uniform in $(x,i)\in\mathbb{R}^{n}\times{\mathbb{Z}}_{+}$. Moreover, it is assumed that $q_{ij}(x)=0$ if $|i-j|<k$ for some constant $k$. It can be seen that our conditions in this paper are much more relaxed compared with the aforementioned reference.

6 Further Remarks

This paper has been devoted to modeling and analysis of switching diffusions in which past-dependent switching processes and countable switching sets are considered. Many problems remain open. Based on our results, one may consider such properties as recurrence, ergodicity, and stability. Future work may also be directed to the study of switching diffusions in which the drift and diffusions are also past dependent. It is important to work out all the details of the control problems presented in the previous sections, which may open up a new avenue for investigation of a wide range of control and optimization problems involving switching diffusions that are treated in this paper.

Appendix A Appendix

This section is devoted to the proofs of some technical results. To simplify the notation, we denote by $\mathbb{P}_{\phi,i}$ the probability measure conditional on the initial value $(\phi,i)$, that is, for any $t>0$,

[TABLE]

[TABLE]

and

[TABLE]

Let $\mathbb{E}_{\phi,i}$ be the expectation associated with $\mathbb{P}_{\phi,i}$. First, we prove the following result.

Lemma A.1**.**

Let either Assumption 3.1 or Assumption 3.3 combined with (ii) of Assumption 3.1 be satisfied. Assume further that $q_{i}(\cdot),q_{ij}(\cdot)$ are continuous functions in $\mathcal{C}$ for each $i,j\in{\mathbb{Z}}_{+}$. Then the solution $(X_{t},\alpha(t))$ to (2.3) satisfies (2.1) and (2.2).

Proof.

It is clear that the solution $(X_{t},\alpha(t))$ to (2.3) satisfies (2.1). Fix $\phi\in\mathcal{C},$ $i,j\in{\mathbb{Z}}_{+},i\neq j$. Applying the generalized Itô formula to the function $V(\psi,k)=0$ if $k\neq j$ and $V(\psi,j)=1$ we have

[TABLE]

where $q_{ii}(\phi):=-q_{i}(\phi)=-\sum_{j\neq i}q_{ij}(\phi)$. 111 $q_{ii}(\phi)$ is not defined in the journal version (SICON) of this paper Since $\alpha(t)$ is cadlag and $X(t)$ is continuous, $\lim_{t\to 0^{+}}\alpha(t)=i$ and $\lim_{t\to 0^{+}}X_{t}=\phi$ $\mathbb{P}_{\phi,i}$-a.s. In light of the continuity of $q_{ij}(\cdot)$ we obtain $\lim_{t\to 0^{+}}q_{\alpha(t),j}(X_{t})=q_{ij}(\phi)\,\,\mathbb{P}_{\phi,i}-\text{a.s.}$ which implies that

[TABLE]

Since $q_{ij}(\cdot)$ is uniformly bounded, so is $\displaystyle\dfrac{1}{\Delta}\int_{0}^{\Delta}q_{\alpha(t),j}(X_{t})dt$. By virtue of the Lebegue dominated convergence theorem, we have

[TABLE]

In the same manner, applying the generalized Itô formula to the function $V(\psi,k)=1$ if $k\neq i$ and $V(\psi,i)=0$, we obtain that

[TABLE]

The proof is complete by noting that (2.2) follows from (A.1) and (A.2) and the Markov property of $(X(t),\alpha(t))$. ∎

Next, we provide the proofs of some results in Section 4.

Proof of Lemma 4.2.

To prove claim (i), we apply the generalized Itô formula to $V(j)=1$ if $j=i$, and $V(j)=0$ if $j\neq i$. We have

[TABLE]

Since $W(\cdot)$ is independent of the Poisson random measure, taking the conditional expectation with respect to $\mathcal{F}^{W}_{T}$ yields

[TABLE]

Hence,

[TABLE]

Since \mathbb{E}_{\phi,i}\big{[}\boldsymbol{1}_{\{\lambda_{1}>0\}}\big{|}\mathcal{F}^{W}_{T}\big{]}=1, we obtain

[TABLE]

Now we prove claim (ii). First, we try to find the distribution of $(\lambda_{1},\beta_{1})$ conditioned on $\mathcal{F}_{T}^{W}$ when $\lambda_{1}\in[0,T]$. Fix $j\neq i$ and let $f(t,k):[0,T]\times{\mathbb{Z}}_{+}\to{\mathbb{Z}}_{+}$ be any bounded measurable function satisfying $f(t,k)=0$ if $k\neq j$. Applying the generalized Itô formula, we obtain

[TABLE]

Since $W(\cdot)$ is independent of the Poisson random measure, taking the conditional expectation with respect to $\mathcal{F}^{W}_{T}$, we have

[TABLE]

As a result, for $t\in[0,T]$,

[TABLE]

Thus,

[TABLE]

as desired. ∎

Proof of Proposition 4.3.

First, we prove (4.2) for the case $l=0$. Since $(X_{t},\alpha(t))=(Y_{t},\lambda(t))$ up to the moment $\alpha_{1}=\lambda_{1}$, we have

[TABLE]

where the last equality is consequence of (i) of Lemma 4.2. Since $\gamma(\cdot)$ and $Y(\cdot)$ are independent, $Z_{t}=Y_{t}$ up to the moment $\theta_{1}$ and $\mathbb{P}_{\phi,i}\{\theta_{1}>T\}=\exp(-T)$, we obtain

[TABLE]

From (A.4) and (A.5), we have for $t\in[0,T]$ that

[TABLE]

We now prove (4.2) for $l=1$. Let $g(\phi,t,i)$, $\tilde{g}(\phi,t,i):\mathcal{C}\times[0,\infty)\times{\mathbb{Z}}_{+}\to\mathbb{R}$ be bounded measurable functions and $g(\phi,t,i)=\tilde{g}(\phi,t,i)=0$ if $t>T$. It follows from (ii) of Lemma 4.2 that

[TABLE]

On the other hand,

[TABLE]

Substituting $\tilde{g}(\phi,t,i)=g(\phi,t,i)\exp(t)\times\dfrac{q_{ii}(\phi)}{\rho_{ii}}$ into (A.8), we have

[TABLE]

It follows from (A.7) and (A.9) that

[TABLE]

We now use the strong Markov property of $(X_{t},\alpha(t))$ and $(Z_{t},\gamma(t))$, (A.10) as well as (A.6) with $\phi,i,T$ replaced by $\phi_{1},i_{1},T-t$, respectively;

[TABLE]

We have already proved (4.2) for $l=0,1$. Using the same argument, the induction, and the strong Markov property of $(X_{t},\alpha(t))$ and $(Z_{t},\gamma(t))$, we can obtain (4.2) for any $l\in{\mathbb{Z}}_{+}$. ∎

The proof of Lemma 4.5.

By (3.5), we can find $m\in{\mathbb{Z}}_{+}$ such that

[TABLE]

Now, let $\varepsilon=\varepsilon(\Delta)>0$ (to be specified later). In view of [14, Theorem 4.3, p. 61], for each $i\in{\mathbb{Z}}_{+}$, there is a constant $C_{i}$ such that

[TABLE]

By the Kolmogorov-Centsov theorem (see [8, Theorem 2.8]), there is a positive random variable $h^{\phi}_{i}(\omega)$ such that

[TABLE]

Since $C_{i}$ in (A.13) does not depend on $\phi\in\{\psi:\|\psi\|\leq R+1\}$, it can be seen from the proof of the Kolmogorov-Centsov theorem that for any $\varepsilon>0$, there is a constant $h_{i}>0$ satisfying

[TABLE]

Without loss of generality, we can choose $h_{i+1}<h_{i},\forall i\in{\mathbb{Z}}_{+}$. Let

[TABLE]

and

[TABLE]

Hence $\mathcal{H}_{i+1,T}\supset\mathcal{H}_{i,T}$ and $\mathcal{H}_{i+1}\supset\mathcal{H}_{i}$. For $d>0$ and a compact set $\mathcal{K}\subset\mathcal{C}$, we define

[TABLE]

Define $\mathcal{K}^{0}=\{\psi(\cdot)=\phi_{0}(\cdot)+c:c\in\mathbb{R}^{n},|c|\leq 1\}$, which is compact, and $\mathcal{K}^{1}=\mathcal{K}^{0}\uplus\mathcal{H}_{i_{0}}.$ For each $\phi\in\mathcal{K}^{1}$, there is $n_{\phi,i_{0}}>i_{0}$ such that

[TABLE]

By the continuous of $q_{i_{0}}$ and $q_{i_{0}k}(\phi)$, there is a $d_{\phi,i_{0}}>0$ such that

[TABLE]

Since $\mathcal{K}^{1}$ is compact, there exist $n_{1}>0$ and $d_{1}>0$ such that

[TABLE]

Define $\mathcal{K}^{2}=\mathcal{K}^{1}\uplus\mathcal{H}_{n_{1}}.$ Using the compactness of $\mathcal{K}^{2}$, there exist $n_{2}>n_{1}$ and $d_{2}\in(0,d_{1}]$ such that

[TABLE]

Continuing this way, for $\mathcal{K}^{m}=\mathcal{K}^{m-1}\uplus\mathcal{H}_{n_{m-1}},$ there exists $n_{m}>n_{m-1}$ and $d_{m}\in(0,d_{m-1}]$ such that

[TABLE]

Set $\mathcal{K}^{\phi,1}=\{\phi\}\uplus\mathcal{H}_{i_{0}}$ and $\mathcal{K}^{\phi,k}=\mathcal{K}^{\phi,k-1}\uplus\mathcal{H}_{n_{k-1}}$ for $\phi\in\mathcal{C}$ and $k=2,\dots,m$. It is not difficult to verify that

[TABLE]

Denote by $\{Y(\cdot)\in\mathcal{H}_{n_{0},T}\}$ the event $\{t\in[0,T]\mapsto Y(t)$ is a function belonging to $\mathcal{H}_{n_{0},T}\}$. Clearly, if $Y(\cdot)\in\mathcal{H}_{i_{0},T}$, then $Y_{t}\in K^{\phi,1}\,\forall t\in[0,T]$. Thus, we can proceed as follows:

[TABLE]

Similarly, if $(\phi_{1},i_{1})\in\mathcal{K}^{\phi,1}\times N_{1}$, then \mathbb{P}_{\phi_{1},i_{1}}\big{\{}\tau_{1}\leq T,\big{(}X_{(1)},\alpha_{1}\big{)}\notin\mathcal{K}^{\phi,2}\times N_{n_{2}}\big{\}}\leq(T+1)\varepsilon. Using the strong Markov property of $(X_{t},\alpha(t))$, we obtain

[TABLE]

Continuing this way, we can show for any $k=1,\dots,m$ that

[TABLE]

Consequently,

[TABLE]

Hence, if we choose $\varepsilon=\dfrac{1}{2m(T+1)}\Delta$,

[TABLE]

It follows from (A.12) and (A.18) that

[TABLE]

It is easily verified that if $\omega\in\{\tau_{m+1}>T\}\cap\{\forall k=1,\dots,m:\tau_{k}>T\text{ or }\alpha_{k}\in N_{n_{k}}\}$, then $\alpha(t)\in N_{n_{m}},\forall t\in[0,T]$. The assertion of the lemma is proved. ∎