Time-changed Stochastic Control Problem and its Maximum Principle Theory

TL;DR

This paper develops a maximum principle framework for stochastic control problems involving time-changed processes driven by Lévy noise, including existence and uniqueness results for associated backward stochastic differential equations.

Contribution

It introduces a maximum principle theory for time-changed stochastic control problems with Lévy noise and proves existence and uniqueness of related backward stochastic differential equations.

Findings

Established maximum principle for time-changed stochastic control

Proved existence and uniqueness of time-changed backward SDEs

Provided illustrative examples

Abstract

This paper studies a time-changed stochastic control problem, where the underlying stochastic process is a L\'evy noise time-changed by an inverse subordinator. We establish a maximum principle theory for the time-changed stochastic control problem. We also prove the existence and uniqueness of the corresponding time-changed backward stochastic differential equation involved in the stochastic control problem. Some examples are provided for illustration.

Equations

$$dX(t)$$

$$+{\int }_{|y|<c}\gamma (t,E_t,X(t-),u(t),y)\tilde{N}(d{E}_t,dy),$$

$$J(u)=\mathbb{E}\Big{[}\int_{0}^{T}g(t,E_{t},X(t),u(t))dE_{t}+h(X(T))\Big{]},\ u\in\mathcal{A},$$

$$J(u^{\ast })=\underset{u\in \mathcal{A}}{\sup }J(u).$$

$$dX(t)$$

$$+{\int }_{|y|<c}\gamma (t,E_t,X(t-),u(t),y)\tilde{N}(d{E}_t,dy),$$

$$J(u)=\mathbb{E}\Big{[}\int_{0}^{T}f(t,E_{t},X(t),u(t))dt+\int_{0}^{T}g(t,E_{t},X(t),u(t))dE_{t}+h(X(T))\Big{]},\ u\in\mathcal{A}.$$

$$\text{mathbbm}E{e}^{-\lambda D(t)}=e^{-t\varphi (\lambda )},$$

$$E_t:=\inf \{\tau >0:D(\tau )>t\}.$$

$$\mathbb{E}[E^{n}_{t}]=\mathcal{L}_{s}^{-1}\Big{[}\frac{n!}{s\phi^{n}(s)}\Big{]}(t),n\in\mathbb{N},$$

$$dX(t)=$$

$$+{\int }_{|y|<c}\gamma (t,E_t,X(t-),u(t),y)\tilde{N}(d{E}_t,dy),$$

$${\mathcal{G}}_t=\bigcap _{u>t}\{[{\mathcal{F}}_y:0\le y\le u]\vee \sigma [E_y:y\ge 0]\}.$$

$$\displaystyle\Big{|}b(t_{1},t_{2},x,u)-b(t_{1},t_{2},y,u)\Big{|}^{2}+\Big{|}\sigma(t_{1},t_{2},x,u)-\sigma(t_{1},t_{2},y,u)\Big{|}^{2}$$

$$\displaystyle+\int_{|z|<c}\Big{|}\gamma(t_{1},t_{2},x,u,z)-\gamma(t_{1},t_{2},y,u,z)\Big{|}^{2}\nu(dz)\leq K|x-y|^{2},$$

$$b(t,E_t,X(t),u(t)),\sigma (t,E_t,X(t),u(t)),\gamma (t,E_t,X(t),u(t),y)\in \mathcal{L}({\mathcal{G}}_t),$$

$$X(t)=$$

$$+{\int }_0^t{\int }_{|y|<c}\gamma (t,E_t,X(t-),y)\tilde{N}(d{E}_t,dy),$$

$$F(t,E_t,$$

$$\displaystyle+\int_{0}^{t}\int_{|y|<c}\Big{[}F(s,E_{s},X(s-)+\gamma(s,E_{s},X(s-),y))-F(s,E_{s},X(s-))\Big{]}\tilde{N}(dE_{s},dy)$$

$$+{\int }_0^t{F}_x(s,E_s,X(s-))\sigma (s,E_s,X(s-))d{B}_{E_s},$$

$$L_{1}F(t_1,$$

$$L_{2}F(t_1,$$

$$+$$

$$dX(t)=-\mu (t,E_t,X(t-),u(t))d{E}_t+u(t)d{B}_{E_t}+{\int }_{\mathbb{R}\setminus \{0\}}h(t,z)\tilde{N}(d{E}_t,dz),$$

$$\displaystyle\Big{|}$$

$$\displaystyle+\int_{t}^{T}2(X_{1}(s)-X_{2}(s))\Big{[}-\Big{(}\mu(s,E_{s},X_{1}(s),u_{1}(s))-\mu(s,E_{s},X_{2}(s),u_{2}(s))\Big{)}dE_{s}+\big{(}u_{1}(s)-u_{2}(s)\big{)}dB_{E_{s}}\Big{]}$$

$$\displaystyle\Big{|}$$

$$=$$

$$\le$$

$$=$$

$$\mathbb{E}\Big{[}\Big{|}X_{1}(t)-X_{2}(t)\Big{|}^{2}\Big{]}\leq(2L_{\mu}+L_{\mu}^{2})\mathbb{E}\Big{[}\int_{t}^{T}|X_{1}(s)-X_{2}(s)|^{2}dE_{s}\Big{]}.$$

$${\int }_t^T(X_1(s)-X_2(s))$$

$$=$$

$$\displaystyle\mathbb{E}\int_{0}^{\infty}\Big{|}1_{\{t\leq D(s-)\leq T\}}(X_{1}(D(s-))-X_{2}(D(s-)))\big{(}u_{1}(D(s-))-u_{2}(D(s-))\big{)}\Big{|}^{2}ds$$

$$\displaystyle\leq\mathbb{E}\int_{0}^{\infty}\Big{|}(X_{1}(D(s-))-X_{2}(D(s-)))\big{(}u_{1}(D(s-))-u_{2}(D(s-))\big{)}\Big{|}^{2}ds<\infty,$$

$$\displaystyle\mathbb{E}\int_{t}^{T}(X_{1}(s)-X_{2}(s))\big{(}u_{1}(s)-u_{2}(s)\big{)}dB_{E_{s}}$$

Full text

1. Introduction

Uncertainty is inherent in the real world and changes over time, putting people’s decisions at risk. A decision maker wants to select the best choice among all possible ones. The stochastic control theory serves as a tool to such dynamic optimization problem. The world has witnessed many applications of stochastic control theory in various fields such as biology [16], economics [3], and finance [15].

A well known approach to stochastic control problem is based on the maximum principle method. Such method for Itô diffusion case is first studied by Kushner [8], Bismut [2] and further developed by Bensoussan [1], Peng [14], and others. The jump diffusion case is formulated by Framstad, Øksendal and Sulem [4]. The idea of the maximum principle approach is to formulate a Hamiltonian function and derive the adjoint equations, which involve the backward stochastic differential equation. Under sufficient conditions, the optimal control is the solution of a coupled system of forward and backward stochastic differential equations.

The time-changed stochastic differential equation and its related fractional Fokker-Plank equation have become an indispensable tool in applied scientific areas. An example of time-changed stochastic differential equation is $dX(t)=dB(E_{t})$ where $X(0)=0$ and $\{E_{t},t\geq 0\}$ is the inverse of an $\alpha-stable$ subordinator, see [10]. The sub-diffusion $B(E_{t})$ is governed by time-fractional diffusion equation $\partial_{t}^{\alpha}q(x,t)=\partial^{2}_{x}q(x,t)$. Some time-changed stochastic differential equations are used to describe real world phenomena. For example, quantitative financial analysts exploit the Black-Scholes framework in derivative pricing, in which the stock price is modeled by Brownian motion. However, some stocks are not actively traded thus their prices stay constant for some time periods. Such phenomenon can be modeled by time-changed Brownian motion but not by the standard Brownian motion, see Figure 1. Fruitful studies in this area are available, see [5, 9, 11, 13].

As time-changed stochastic processes have been adopted in more and more areas, we believe it is necessary to study the stochastic control problem based on the time-changed stochastic process, which will build up a framework to solve potential optimization problems. In this paper, we investigate the time-changed stochastic control problem using the maximum principle method. Specifically, we consider the following time-changed stochastic process, see [7, 12]:

[TABLE]

with $X(0)=x_{0}\neq 0$ and the corresponding performance function

[TABLE]

where $u(t)=u(t,w)\in U\subset\mathbb{R}$ is the control and $\mathcal{A}$ denotes the set of $admissible$ controls. We establish a maximum principle theory for the stochastic control problem to find $u^{*}\in\mathcal{A}$ such that

[TABLE]

Then we extend such result to a more general time-changed stochastic process involving time drift term $dt$:

[TABLE]

with $X(0)=x_{0}\neq 0$, and the corresponding performance function

[TABLE]

In the remaining parts of this paper, some necessary concepts and preliminary results will be given in Section 2. In section 3 and 4, we establish a maximum principle theory for time-changed stochastic control problems mentioned above and provide some examples for illustration.

2. Preliminaries

Let $(\Omega,\mathcal{F},(\mathcal{F}_{t}),P)$ be a filtered probability space satisfying usual hypotheses of completeness and right continuity. Assume that an independent $\mathcal{F}_{t}$-adapted Poisson random measure $N$ is defined on $\mathbb{R}_{+}\times(\mathbb{R}-\{0\})$ with compensator $\tilde{N}$ and intensity measure $\nu$, where $\nu$ is a Lévy measure such that $\tilde{N}(dt,dy)=N(dt,dy)-\nu(dy)dt$ and $\int_{\mathbb{R}-\{0\}}(|y|^{2}\land 1)\nu(dy)<\infty$.

Let $\{D(t),t\geq 0\}$ be a right continuous with left limits (RCLL) subordinator starting from 0 with Laplace transform

[TABLE]

where Laplace exponent $\phi(\lambda)=\int_{0}^{\infty}(1-e^{-\lambda x})\nu(dx)$ , define its inverse

[TABLE]

Lemma 2.1**.**

(Lemma 8 in [7]) Let E be the inverse of a subordinator D with Laplace exponent $\phi$ and infinite Lévy measure. Then $\mathbb{E}[e^{\lambda E_{t}}]<\infty$, $\forall\lambda\in\mathbb{R}$ and $t\geq 0$. In particular, for each $t>0$, moments of $E_{t}$ of all orders exist and are given by

[TABLE]

where $\mathcal{L}_{s}^{-1}[g(s)]$ denotes the inverse Laplace transform of a function $g(s)$.

Consider the following time-changed stochastic differential equation:

[TABLE]

with $X(0)=x_{0}\neq 0$, where $b,\sigma,\gamma$ are real-valued functions satisfying the following Lipschitz condition 2.2 and assumption 2.3 such that there exists a unique $\mathcal{G}_{t}$-adapted process $X(t)$ satisfying time-changed SDE (2.4), see Lemma 4.1 in [6]. The filtration $\{\mathcal{G}_{t}\}_{t\geq 0}$ is defined as

[TABLE]

Assumption 2.2**.**

(Lipschitz condition) There exists a positive constant K such that

[TABLE]

for all $t_{1},t_{2}\in\mathbb{R}_{+}$ and $x,y\in\mathbb{R}$.

Assumption 2.3**.**

If $X(t)$ is a RCLL and $\mathcal{G}_{t}$-adapted process, then

[TABLE]

where $\mathcal{L}(\mathcal{G}_{t})$ denotes the class of left continuous with right limits (LCRL) and $\mathcal{G}_{t}$-adapted processes.

The process $u(t)=u(t,w)\in U\subset\mathbb{R}$ is the control. Assume that $u$ is adapted and RCLL, and that the corresponding equation (2.4) has a unique strong solution $X^{(u)}(t),t\in[0,T]$. Such controls are called $admissible$. The set of admissible controls is denoted by $\mathcal{A}$.

Lemma 2.4**.**

(Itô Formula for Time-Changed Lévy Noise, Lemma 3.1 in [12]) Let $D(t)$ be a RCLL subordinator and $E_{t}$ its inverse process as (2.2). Let $X$ be a process defined as following:

[TABLE]

*where $\mu,b,\sigma,\gamma$ are measurable functions such that all integrals are defined. Here $c$ is the maximum allowable jump size.

Then, for all $F:\mathbb{R}_{+}\times\mathbb{R}_{+}\times\mathbb{R}\rightarrow\mathbb{R}$ in $C^{1,1,2}(\mathbb{R}_{+}\times\mathbb{R}_{+}\times\mathbb{R},\mathbb{R})$, with probability one,*

[TABLE]

where

[TABLE]

Lemma 2.5**.**

(Existence and Uniqueness of BSDE)

Consider the following time-changed Backward stochastic differential equation

[TABLE]

with $X(T)=X$, where $\mu\in L^{2}(\mathbb{R}_{+},\mathbb{R}_{+},\mathbb{R},\mathbb{R}),h\in L^{2}(\mathbb{R}_{+},\mathbb{R})$. If there exists a positive constant $L_{\mu}>0$ such that |\mu(t_{1},t_{2},x_{1},u_{1})-\mu(t_{1},t_{2},x_{2},u_{2})|\leq L_{\mu}\Big{(}|x_{1}-x_{2}|+|u_{1}-u_{2}|\Big{)}, then there exists a unique solution $(X(t),u(t))$ of (2.11).

Proof.

To prove the uniqueness, suppose $(X_{1}(t),u_{1}(t))$ and $(X_{2}(t),u_{2}(t))$ are two solutions to (2.11) in $L^{2}(\Omega\times\mathbb{R}_{+})\times L^{2}(\Omega\times\mathbb{R}_{+})$. By Itô formula,

[TABLE]

Thus,

[TABLE]

Take expectations on both sides,

[TABLE]

Note that we apply Martingale property to derive inequality (2.14) and lay some details below.

[TABLE]

since $(X_{1}(t),u_{1}(t))$ and $(X_{2}(t),u_{2}(t))$ are in $L^{2}(\Omega\times\mathbb{R_{+}})$,

[TABLE]

we have

[TABLE]

Next we apply time-changed Gronwall’s method by Lemma 3.1 in [17]. Define $F(t)=\int_{t}^{T}|X_{1}(s)-X_{2}(s)|^{2}dE_{s}$, then $F(T)=0$ and

[TABLE]

thus

[TABLE]

Taking expectations and letting $k=2L_{\mu}+L_{\mu}^{2}$ imply that

[TABLE]

It follows that

[TABLE]

so $X_{1}(s)=X_{2}(s)$ a.s. for $\forall s\in(t,T)$. By (2.12), since $X_{1}(s)=X_{2}(s)$ a.s. for $\forall s\in(t,T)$, we have $\int_{t}^{T}|u_{1}(s)-u_{2}(s)|^{2}dE_{s}=0$, thus $u_{1}(s)=u_{2}(s)$ a.s. for $\forall s\in(t,T)$. The uniqueness is proved.

To prove the existence, let $u_{0}(t)=0$, $\{(X_{n}(t),u_{n}(t));0\leq t\leq T\}_{n\geq 1}$ be a sequence defined recursively by

[TABLE]

Then

[TABLE]

By Itô formula in Lemma 2.4, there exists $k>0$ such that

[TABLE]

Taking expectation on both sides implies

[TABLE]

Define F_{n}(t)=\int_{t}^{T}\Big{|}X_{n}(s)-X_{n-1}(s)\Big{|}^{2}dE_{s} for all $n\geq 1$, then $F_{n}(T)=0$ and

[TABLE]

By a similar argument for uniqueness and using (2.25),

[TABLE]

letting $t=0$,

[TABLE]

Thus, $\{X_{n}\}$ is a Cauchy sequence in $L^{2}(\Omega\times\mathbb{R}_{+})$. Taking (2.25) into consideration, $\{u_{n}\}$ is also a Cauchy sequence in $L^{2}(\Omega\times\mathbb{R}_{+})$. Thus, the existence of solution to (2.11) is proved. ∎

3. Time-changed Stochastic Control Problem

In this section, we solve the time-changed stochastic control problem through the maximum principle approach. An example is provided to illustrate how our method works for a particular time-changed stochastic problem.

We consider a performance criterion $J=J(u)$ of the form

[TABLE]

where $g:[0,T]\times\mathbb{R}_{+}\times\mathbb{R}\times U\rightarrow\mathbb{R}$ is continuous, $h:\mathbb{R}\rightarrow\mathbb{R}$ is $C^{1},T<\infty$ is a fixed deterministic time and

[TABLE]

The stochastic control problem is to find the optimal control $u^{*}\in\mathcal{A}$ such that

[TABLE]

Since $E_{t}$ is right continuous and nondecreasing, $\frac{dE_{t}}{dt}$ exists for $t\geq 0$ a.e.

Define the $Hamiltonian\ H:[0,T]\times\mathbb{R_{+}}\times\mathbb{R}\times U\times\mathbb{R}\times\mathbb{R}\times\mathcal{R}$ by

[TABLE]

or

[TABLE]

where $\mathcal{R}$ is the set of functions $r:\mathbb{R_{+}}\times\mathbb{R}\to\mathbb{R}$ such that the integrals in (3.4) exists.

Define the adjoint equation in the unknown processes $p(t)\in\mathbb{R},\ q(t)\in\mathbb{R}$, and $r(t,z)\in\mathbb{R}$ in the backward stochastic differential equations

[TABLE]

Theorem 3.1**.**

(Time-Changed Maximum Principle Theorem) Let $\hat{u}\in\mathcal{A}$ with corresponding solution $\hat{X}=X^{(\hat{u})}$ of (2.4) and suppose there exists a solution $(\hat{p}(t),\hat{q}(t),\hat{r}(t,z))$ of the corresponding adjoint equation (3.6) satisfying

[TABLE]

and

[TABLE]

Moreover, suppose that

[TABLE]

for all $t$, that $h(x)$ in (3.1) is a concave function of $x$ and that

[TABLE]

exists and is a concave function of $x$ for all $t\in[0,T]$. Then $\hat{u}$ is an optimal control of stochastic control problem (3.3).

Proof.

Let $u\in\mathcal{A}$ be an admissible control with corresponding state process $X(t)=X^{(u)}(t)$. We would like to show that

[TABLE]

Since $g$ is concave, using Itô formula (2.9),

[TABLE]

Among above terms,

[TABLE]

Thus,

[TABLE]

In addition,

[TABLE]

and by (3.6) we have

[TABLE]

Then, since $H$ is concave in $x$, putting equations (3.15) and (3.16) into (3.14) and following the proof in [4], we get

[TABLE]

∎

Example 3.2**.**

*(The Time-Changed Stochastic Linear Regulator Problem)

The Linear Regulator Problem aims to reduce the amount of work or energy consumed by the control system to optimize the controller. In this example, we consider the following time-changed stochastic linear regulator problem:*

[TABLE]

where

[TABLE]

Construct the $Hamiltonian$:

[TABLE]

The adjoint equations are

[TABLE]

The first and second order condition implies that $Hamiltonian:\ H(t_{1},t_{2},x,u,p,q,r)$ achieves the minimum at $u^{*}(t)=-p(t)$.

To find an explicit solution of $u^{*}(t)$, suppose $p(t)=h(E_{t})X(t)$, where $h:\mathbb{R}_{+}\rightarrow\mathbb{R}_{+}$. Then $u^{*}(t)=-h(E_{t})X(t)$ and

[TABLE]

Compare (3.21) and (3.22), $-h^{2}(E_{t})+h^{\prime}(E_{t})=-1$ and $h(E_{T})=2\lambda$. The general solution to this ordinary differential equation gives

[TABLE]

Thus, we have the explicit formula for the optimal control $u^{*}(t)=-h(E_{t})X(t)$. Similarly, $q(t)=h(E_{t})\sigma$ and $r(E_{t},z)=h(E_{t})z$. A simulation of the optimal control $u^{*}(t)$ with $\lambda=-\frac{1}{2},\sigma=1,x_{0}=-.01$, standard normal distribution $\nu$, and inverse stable subordinator $E(t)$ having $\alpha=.9$ is displayed in Figure 2.

Keeping all others parts the same as in the figure 2, we also simulate the optimal control $u^{*}(t)$ for $\alpha=.7$ and $\alpha=.5$ in Figure 3 and 4, respectively. Overall, replacing $t$ by $E_{t}$ would only insert some constant periods into the original process. As $\alpha$ gets closer to $1$, the constant periods vanish gradually.

4. A More General Time-changed Stochastic Control Problem

Now we extend the time-changed SDE (2.4) to a more general case by adding a time drift term as below,

[TABLE]

with $X(0)=x_{0}\neq 0$, where $\mu,b,\sigma,\gamma$ are real-valued functions satisfying the Lipschitz condition 2.2 and assumption 2.3.

Suppose the performance function is given by

[TABLE]

where the function $f,g:[0,T]\times\mathbb{R}_{+}\times\mathbb{R}\times U\rightarrow\mathbb{R}$ are continuous, $h:\mathbb{R}\rightarrow\mathbb{R}$ is $C^{1},T<\infty$ is a fixed deterministic time and

[TABLE]

The stochastic control problem is to find the optimal control $u^{*}\in\mathcal{A}$ such that

[TABLE]

Remark 4.1**.**

Performance functions (3.1) and (4.2) are slightly different in terms of their integral kernels. This difference results in different $Hamiltonians$ and adjoint equations.

Define the $Hamiltonian\ H\ :[0,T]\times\mathbb{R_{+}}\times\mathbb{R}\times U\times\mathbb{R}\times\mathbb{R}\times\mathcal{R}\to\mathbb{R}$ by

[TABLE]

or

[TABLE]

Define the adjoint equation

[TABLE]

Theorem 4.2**.**

(Time-Changed Maximum Principle Theorem) Let $\hat{u}\in\mathcal{A}$ with corresponding solution $\hat{X}=X^{(\hat{u})}$ and suppose there exists a solution $(\hat{p}(t),\hat{q}(t),\hat{r}(t,z))$ of the corresponding adjoint equation (3.6) satisfying

[TABLE]

and

[TABLE]

Moreover, suppose that

[TABLE]

for all $t>0$, that $h(x)$ in (4.2) is a concave function of $x$ and that

[TABLE]

exists and is a concave function of $x$ for all $t\in[0,T]$. Then $\hat{u}$ is an optimal control of stochastic control problem (4.4).

Proof.

Let $u\in\mathcal{A}$ be an admissible control with the corresponding state process $X(t)=X^{(u)}(t)$. We would like to show that

[TABLE]

Since $h$ is concave, using Itô formula (2.9),

[TABLE]

Among above terms,

[TABLE]

Thus,

[TABLE]

In addition,

[TABLE]

and

[TABLE]

Then, by concavity of $H$ and following the proof in [4],

[TABLE]

∎

Example 4.3**.**

(Income and Consumption Optimization) Consider the stochastic control problem

[TABLE]

where

[TABLE]

and

[TABLE]

where $\delta>0,\sigma,$ and $\theta$ are constants and $b=-\frac{\sigma^{2}+\theta^{2}\int_{\mathbb{R}}z^{2}\nu(dz)}{2}$.

We can interpret $u(t)$ as the consumption rate, $X(t)$ as the corresponding wealth, and $\tau$ as the bankruptcy time. Then $\Phi$ represents the maximal expected total quadratic utility of the consumption up to bankruptcy time.

Define the $Hamiltonian\ H:$

[TABLE]

and the adjoint equation

[TABLE]

Let $\frac{\partial H}{\partial u}=(-p(t)+2u(t)\exp(-\delta t))=0$, we have $u^{*}(t)=\frac{p(t)}{2}\exp(\delta t)$. Suppose that $p(t)=h(t)X(t)$, then $u^{*}(t)=\frac{h(t)X(t)}{2}\exp(\delta t)$, thus

[TABLE]

Comparing (4.23) and (4.24), we derive that $h^{\prime}(t)=\frac{h(t)}{2}e^{\delta t}$, equivalently, $h(t)=\exp(\frac{1}{2\delta}e^{\delta t}),$ thus

[TABLE]

Moreover,

[TABLE]

Some algebra implies that

[TABLE]

A simulation of the optimal control $u^{*}(t)$ with $\delta=-.001,\sigma=1,\theta=1,x_{0}=1$, standard normal distribution $\nu$, and inverse stable subordinator $E(t)$ having $\alpha=.9$ is displayed in Figure 5.

Because of the existence of $dt$ term in the underlying process $X(t)$, the simulated process $u^{*}(t)$ has no periods of constant value. Compared with $dE_{t}$ terms, $dt$ term plays the dominating role in the evolution of corresponding wealth $X(t)$, see [11] for a detailed discussion. More specifically, the increasing trend $bX(t)dE_{t}$ is dominated by the consumption rate $-u(t)dt$. Consequently, the optional consumption rate declines as the wealth shrinks in the long term.