Optimal Control of Markov Regime-Switching Stochastic Recursive Utilities

TL;DR

This paper develops a stochastic maximum principle for optimal control of systems with Markov regime-switching and recursive utilities, addressing non-convex control domains and control-dependent diffusion terms.

Contribution

It introduces new first and second-order adjoint equations and derives variational equations for forward-backward stochastic differential equations in this context.

Findings

Established a general maximum principle for complex stochastic control systems.

Derived novel second-order adjoint equations involving solutions of second-order adjoint.

Provided illustrative examples demonstrating the theoretical results.

Abstract

In this paper, we establish a general stochastic maximum principle for optimal control for systems described by a continuous-time Markov regime-switching stochastic recursive utilities model. The control domain is postulated not to be convex, and the diffusion terms depend on control variables. To this end, we first study a kind of classical forward stochastic optimal control problems. Afterwards, based on previous results, we introduce two groups of new first and second-order adjoint equations. The corresponding variational equations for forward-backward stochastic differential equations are derived. In particular, the generator in the maximum principle contains solutions of second-order adjoint equation which is novel. Some interesting examples are concluded as well.

Equations

$$\alpha (t)=\alpha (0)+{\int }_0^t{Q}^{\top }\alpha (s)\mathrm{d}s+M(t),$$

$${\mathcal{J}}_{ij}(t)$$

$$m_{ij}={\int }_0^t⟨\alpha (s-),e_i⟩⟨\mathrm{d}M(s),e_j⟩\mathrm{d}s$$

$${\Phi }_j(t)$$

$${\lambda }_j(t)=\sum _{i=1,i\ne j}^D{\lambda }_{ij}{\int }_0^t⟨\alpha (s-),e_i⟩\mathrm{d}s.$$

$$x(t)$$

$$y(t)$$

$$b$$

$$\sigma$$

$$\gamma$$

$$f$$

$$g$$

$$\mathcal{U}_{ad}=\left\{u(\cdot):[0,T]\times\Omega\rightarrow U\left|\begin{array}[]{l}u(\cdot)\text{ is an }\mathcal{F}_{t}\text{-adapted process with values in }U\\ \text{satisfying }\mathbb{E}\Big{[}\int_{0}^{T}|u(s)|^{\beta}\mathrm{d}s\Big{]}<0\end{array}\right.\right\}.$$

$$J(u(\cdot ))=y(0).$$

$${\mathcal{S}}^2(0,T;{\mathbb{R}}^L)\triangleq$$

$${\mathcal{M}}^2(0,T;{\mathbb{R}}^{L\times d})\triangleq$$

$${\mathcal{V}}^2(0,T;{\mathbb{R}}^{L\times d})$$

$$|f(t,x,y,z,\kappa ,u,e_i)-f(t,x^{\prime },y^{\prime },z^{\prime },{\kappa }^{\prime },u,e_i)|$$

$$|b(t,x,u,e_i)|+|\sigma (t,x,u,e_i)|+|f(t,x,y,z,\kappa ,u,e_i)|$$

$$\mathrm{d}x(t)$$

$$\phi (T,x(T),\alpha (T))-\phi (0,x(0),\alpha (0))$$

$$\mathrm{d}{X}_i(t)={\gamma }_i(t,\alpha (t))\mathrm{d}\tilde{\Phi }(t),i=1,2$$

$$X_1(t)X_2(t)$$

$$\mathrm{d}(X_1(t)X_2(t))$$

$$[X_1,X_2](t)$$

$$\left\{\begin{array}[]{rcl}-\mathrm{d}y(t)&=&[A(t)y(t)+B(t)z(t)+C(t)\kappa(t)+F(t,\alpha(t-))]\mathrm{d}t\\ &&-z(s)\mathrm{d}W(s)-\kappa(s)\mathrm{d}\tilde{\Phi}(s),\\ y(T)&=&\xi,\end{array}\right.$$

$$\left\{\begin{array}[]{l}|A(t)|,\text{ }|B(t)|,\text{ }|C(t)|\leq K,\quad\text{a.e. }t\in[0,T],\text{ }P\text{-a.s.},\\ \displaystyle\int_{0}^{T}[\mathbb{E}|F(s,a(s-))|^{2k}]^{\frac{1}{2k}}\mathrm{d}s<\infty,\end{array}\right.$$

$$\displaystyle\sup_{0\leq t\leq T}\mathbb{E}\big{[}|y(s)|^{2k}\big{]}+\mathbb{E}\bigg{[}\int_{0}^{T}\Big{[}|z(s)|^{2}+\sum_{j=1}^{D}|\kappa_{j}(s)|^{2}\lambda_{j}(s)\Big{]}\mathrm{d}s\bigg{]}$$

$$u^{\varepsilon}(t)=\left\{\begin{array}[]{ll}v,&\text{if }\tau\leq t\leq\tau+\varepsilon,\\ \bar{u}(t),&\text{otherwise,}\end{array}\right.$$

$$\mathcal{J}(u(\cdot))=\mathbb{E}\bigg{[}\int_{0}^{T}l(t,x(t),u(t),\alpha(t-))\mathrm{d}t+h(x(T),\alpha(T))\bigg{]},$$

$$\mathcal{J}(\bar{u}(\cdot ))=\underset{u(\cdot )\in {\mathcal{U}}_{ad}}{\inf }\mathcal{J}(u(\cdot )).$$

$$J(u^{\epsilon }(\cdot ))-J(\bar{u}(\cdot ))\ge 0.$$

$$\phi (t)$$

$${\phi }_x(t)$$

Full text

AMS subject classifications: 93E20, 60H15, 60H30.

**Key words: ** Stochastic maximum principle, Regime-switching, Adjoint equations, Spike variation.

1 Introduction

A large number of literatures focus on applications of Markov regime-switching models in finance as well as stochastic optimal control since the fundamental feature brings from the Markov regime-switching models in contrast to the traditional models described via the diffusion processes from the empirical point of view. The basic idea of regime-switching is to modulate the model with a continuous-time and finite-state Markov chain where each state represents a regime of the system or level of the economic indicator. The regime-switching models have been variously applied in stock trading, option pricing, portfolio selection, risk measurement, etc (see [21, 29, 35, 36, 32] references therein).

It is well known that stochastic optimal control is one of the central themes of modern control sciences. Necessary conditions for the optimal control of the forward continuous-time stochastic system, which are the so-called stochastic maximum principle of Pontraygin’s type, have been extensively studied since early 1960s. As a matter of fact, the basic idea of the stochastic maximum principle is to derive a set of necessary conditions that must be fulfilled by any optimal control. The stochastic maximum principle describes that any optimal control must satisfy a system of forward-backward stochastic differential equations (FBSDEs, for short), called the optimality system, and minimize a functional, called the Hamiltonian. It leads to explicit expressions for the optimal controls in certain cases. The stochastic maximum principle can be applied to situations where state processes involve random coefficients and state constraints. Recently, there are some works applying the stochastic maximum principle in finance (see Cadenillas and Karatzas [4], see references therein).

When Brownian motion is the unique noise source, Peng [28] initially studied a general stochastic maximum principle using the second-order adjoint equations, which allow the control to enter into both drift and diffusion coefficients under a nonconvex control domain. For more details about the stochastic maximum principle theory, interested readers may refer to Yong and Zhou [9]. Forward-backward stochastic control systems driven by FBSDEs are widely applied in mathematical economics and mathematical finance. They mainly emerge in stochastic recursive utility optimization problems and principal-agent problems. Pardoux and Peng [25] proved the well-posedness for nonlinear BSDE. Duffie and Epstein [7] introduced the notion of recursive utilities in continuous-time situation, which is actually a type of backward stochastic differential equation (BSDE, for short) where the generator $f$ is independent of $z$. El Karoui et al [19] extended the recursive utility to the case where $f$ contains $z$. The term $z$ can be interpreted as an ambiguity aversion term in the market (see Chen and Epstein 2002, [2]). In particular, the celebrated Black-Scholes formula indeed provides an effective way of representing the option price (which is the solution to a kind of linear BSDE) through the solution to the Black-Scholes equation (parabolic partial differential equation actually). Since then, BSDE has been extensively studied and employed in the areas of applied probability and optimal stochastic controls, particularly in financial engineering (see [19]).

Peng [28] derived necessary conditions for optimal control for the partially coupling case when the control domain is convex. Later, Xu [42] studied the nonconvex control domain case and obtained the corresponding necessary conditions. In the above work, however, assume that the diffusion term in the forward control system does not include the control variable. Ji and Zhou applied the Ekeland variational principle to establish a maximum principle for a partially coupled forward-backward stochastic control system, while the forward state variable is constrained in a convex set at the terminal time. Meng [24] and Wu [41] obtained the necessary conditions for optimal control of fully coupled forward-backward stochastic control systems when the control domain is convex. Shi and Wu [31] studied the nonconvex control domain case and obtained the corresponding necessary conditions under some $G$-monotonic assumptions. However, the control variable is not involved in the diffusion term of the forward equation.

In order to study the backward linear-quadratic optimal control problem, Kohlmann and Zhou [20], Lim and Zhou [22] developed a kind of new method to handle the problem. The variable $z$ is regarded as a control process and the terminal condition $y_{T}=h(x_{T})$ as a constraint, and then it is possible to apply the Ekeland variational principle to obtain the maximum principle. Adopting this idea, Yong [44] and Wu [40] independently established the maximum principle for the recursive stochastic optimal control problem. Nonetheless, the maximum principle derived by these method involves two unknown parameters. Hence, some hard questions naturally emerge as follows: What is the second-order variational equation for the BSDE? How to derive the second-order adjoint equation since the quadratic form with respect to the variation of $z$. All of which seem to be extremely complicated. Hu [14] overcame the above difficulties by introducing two new adjoint equations. Then, the second-order variational equation for BSDE and the maximum principle are obtained. The main difference of his variational equations compared with those in Peng [37] consists in the term $\langle p(t),\delta\sigma(t)\rangle I_{E_{\varepsilon}}(t)$ in the variation of $z$. Due to the term $\langle p(t),\delta\sigma(t)\rangle I_{E_{\varepsilon}}(t)$ in the variation of $z$, Hu developed a global maximum principle, which is novel and different from those in Wu [40], Yong [44] and other previous works, to solve Peng’s open problem. Hu, Ji and Xue further considered the maximum principle for fully coupled FBSDEs (see [15, 16]).

Inspired by above works, we are more interested in studying optimal control for FBSDE of Markov-regime diffusion type. Compared with the literature mentioned above, the novelty of the formulation and the contribution in this paper should be stated as follows: This work states the necessary condition for stochastic utilities, the maximum principle of ours is significantly different from the existing results obtained in [8, 45, 32] which describe the sufficient condition. In particular, Sun et al [33] considered the forward-backward controlled systems and established a sufficient condition for optimal control. This paper fills a much-needed gap for the necessary part. Due to the presence of regime-switching, the second-order solution enters in the generator of BSDE which is new. Tao and Wu [34] also consider optimal control for FBSDEs modulated by continuous-time and finite-state Markov chains with convex control domain (see [23, 26]). Recently, a similar work [49] has been done by Zhang et al for forward mean field system with jump, but not containing the generator. In this paper, we adopt the idea from Peng [37] within our own framework using the Riesz Representation Theorem to interpret the dual relationship between the adjoint and variational equation, which is much clearer and more readable. Besides, we prove a general estimate for BSDE (see Lemma 4). Then, we can easily derive some delicate estimates for backward variational equation which is important to establish the variational inequality.

The rest of this paper is organized as follows: After some preliminaries and notation in the second section, we are devoted the third section to studying the classical forward stochastic control problems adopting the form, similar to [37]. Based on the previous estimates, we establish a general maximum principle for FBSDEs optimal control problem. Finally, in Section 5, we conclude the novelty of this paper and schedule possible generalizations in future. A proof of technique lemma is put in the Appendix 5.

2 Preliminaries

Throughout this paper, we denote by $\mathbb{R}^{n}$ the space of $n$-dimensional Euclidean space, by $\mathbb{R}^{n\times d}$ the space the matrices with order $n\times d$. Let $(\Omega,\mathcal{F},P)$ be a completed filtered probability space. For a fixed finite time $T$ horizon, $\{W_{t},0\leq t\leq T\}$ is an $\mathbb{R}^{d}$-valued standard Brownian motion defined on the space.

To model the controlled state process, we first introduce a set of Markov jump martingales associated with the chain $\alpha$. We adopt the notation from Elliott et al [11]. The state space of the chain with a finite state space is denoted by $S:=\{e_{1},e_{2},\ldots,e_{D}\}$, where $D\in N$, $e_{i}\in\mathbb{R}^{D}$, and the $j$-th component of $e_{i}$ is the Kronecker delta $\delta_{ij}$ for each $1\leq i,j\leq D$. The state space $S$ is called a canonical state space. We suppose that the chain is homogeneous and irreducible. To specify statistical or probabilistic properties of the chain $\alpha$, we define the generator $Q:=[\lambda_{ij}]_{i,j=1,2,\ldots,D}$ of the chain under $P$. This is also called the rate matrix, or the $Q$-matrix. For each $i,j=1,2,\ldots,D$, $\lambda_{ij}$ is the constant transition intensity of the chain from state $e_{i}$ to state $e_{j}$ at time $t$. Note that $\lambda_{ij}\geq 0$ for $i\neq j$ an $\sum_{j=1}^{D}\lambda_{ij}=0$, so $\lambda_{ii}\leq 0$. In general, for each $i,j=1,2,\ldots,D$ with $i\neq j$, we suppose that $\lambda_{ij}>0$, which means that $\lambda_{ii}<0$. From Elliott et al [11], we have the following semimartingale expression for $\alpha$:

[TABLE]

where $\{M(t)\}_{0\leq t\leq T}$ is an $\mathbb{R}^{D}$-valued, $(\mathcal{F},P)$-martingale. For each $i,j=1,2,\ldots,D$, with $i\neq j$, let $\mathcal{J}_{ij}(t)$ be the number of jumps from state $e_{i}$ to state $e_{j}$ up to time $t$. Then

[TABLE]

where

[TABLE]

is an $(\mathcal{F},P)$-martingale. The $m_{ij}$’s are called the basic martingales associated with the chain $\alpha$. For each fixed $j=1,2,\ldots,D$, let $\Phi_{j}(t)$ be the number of jumps into state $e_{j}$ up to time $t$. Thus,

[TABLE]

Set $\tilde{\Phi}_{j}(t):=\sum_{i=1,i\neq j}^{D}m_{ij}(t)$. For each $j=1,2,\ldots,D$, $\{\tilde{\Phi}_{j}(t)\}_{0\leq t\leq T}$ is also an $(\mathcal{F},P)$-martingale. For convenience, define

[TABLE]

Then, for every $j=1,2,\ldots,D$, $\tilde{\Phi}_{j}(t)=\Phi_{j}(t)-\lambda_{j}(t)$ is also an $(\mathcal{F},P)$-martingale.

Note that $\top$ appearing in this paper as superscript denotes the transpose of a matrix. Let $U$ be a compact subset of $\mathbb{R}^{k}$. In what follows, $K$ represents a generic constant, which can be different from line to line.

The state process is governed by FBSDEs of the following type:

[TABLE]

for all $t\in[0,T]$, $P$-a.s., with $X(t-)=x$, on some filtered probability space $(\Omega,\mathcal{F},P)$, where

[TABLE]

are given deterministic functions. Denote $\mathcal{U}_{ad}$ as the admissible control set, where

[TABLE]

Our problem is to minimize the following cost functional over $\mathcal{U}_{ad}:$

[TABLE]

Our stochastic recursive optimal control problem is to find an optimal control $u(\cdot)\in\mathcal{U}_{ad}$ such that $J(\bar{u}(\cdot))=\inf\limits_{u(\cdot)\in\mathcal{U}_{ad}}y(0)$.

We now introduce the following spaces of processes:

[TABLE]

Set $\mathcal{N}^{2}(0,T)\triangleq\mathcal{S}^{2}(0,T;\mathbb{R}^{L})\times\mathcal{S}^{2}(0,T;\mathbb{R}^{L})\times\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times d})\times\mathcal{V}^{2}(0,T;\mathbb{R}^{L\times d})$. Assume that

(A1)

The coefficients $b$ and $\sigma$ satisfy the Lipschitz condition for $x$, and $f(t,x,y,z,\kappa,u,e_{i})$ satisfies the Lipschitz condition as follows:

[TABLE]

$x,x^{\prime}\in\mathbb{R}^{L},y,y^{\prime},z,z^{\prime}\in\mathbb{R},\kappa,\kappa^{\prime}\in\mathbb{R}^{D}$, for every $e_{i}\in S$. Moreover,

[TABLE]

for all $e_{i}\in S$, where $\|\kappa\|=\sum_{j=1}^{D}|\kappa_{j}|^{2}\lambda_{j}$.

(A2)

Maps $\theta=(x,y,z,\kappa)\rightarrow b(t,x,u,e_{i})$, $\sigma(t,x,u,e_{i})$, $f(t,\theta,u,e_{i})$ is twice continuously differential, with the (partial) derivatives up to the second order being uniformly bounded, Lipschitz continuous in $(x,y,z,\kappa)$, and continuous in $u\in U$ and $e_{i}\in S$. The map $g(x,e_{i})$ is twice differentiable with the derivatives up to the second order being uniformly bounded and uniformly Lipschitz continuous in $e_{i}\in S$.

Clearly, under Assumptions (A1)-(A2), FBSDEs (1) and (2) admit a unique adapted strong solution $(x,y,z,\kappa)\in\mathcal{N}^{2}(0,T)$ (see [10]). Due to the influence of Markov regime switching, we need an extension of Itô’s formula with Markov regime switching in the sequel.

Lemma 1**.**

Assume that an $L$-dimensional process $x(\cdot)$ driven by

[TABLE]

and the function $\varphi(\cdot,\cdot,e_{i})\in C^{1,2}([0,T]\times\mathbb{R}^{L})$ for every $e_{i}\in S$. Then

[TABLE]

where $\gamma^{m}$ denotes the $m$-th columns of the matrix $\gamma$.

Lemma 2**.**

Let

[TABLE]

be one-dimensional regime switching equations. Then

[TABLE]

Proof. Applying Itô’s formula to $Y(t)=X_{1}(t)X_{2}(t)$, we have

[TABLE]

which implies the desired result. $\Box$

Remark 3**.**

The process

[TABLE]

is called the quadratic covariance of $X_{1}$ and $X_{2}$.

We now present a technical result which will be useful in the next section. Its proof can be found in Appendix 5.

Lemma 4**.**

Consider the following linear BSDE with Makov regime switching:

[TABLE]

where $\xi\in L^{2}(\mathcal{F}_{T})$ with $\mathbb{E}[|\xi|^{2}]<\infty$, $A,B:[0,T]\times\Omega\rightarrow\mathbb{R}$, $C:[0,T]\times\Omega\rightarrow\mathbb{R}^{D}$ and $F:[0,T]\times S\times\Omega\rightarrow\mathbb{R}$ are $\mathcal{F}_{t}$-adapted, and

[TABLE]

for some $k\geq 1$. Then there exists a positive constant $\tilde{K}$ such that

[TABLE]

We outline the proof for the convenience of the reader, which can be found in the Appendix 5.

3 Classical optimal control problems

The aim of this section is to derive a kind of variational equation (first and second-order) and corresponding variational inequality. Observe that the control variable enters in $\sigma$, $\gamma$ and the control domain $U$ is non-convex, both the convex perturbation and the first-order expansion approach fail. In contrast to Peng [37], one needs to find new adjoint equations because of presence of Markov chains. We now introduce the spike variation with respect to optimal control $\bar{u}(\cdot)$, more precisely, let $\varepsilon>0$ and $E_{\varepsilon}\subset[0,T]$ be a Borel set with Borel measure $|E_{\varepsilon}|=\varepsilon$, defined as follows:

[TABLE]

where $0\leq\tau<T$ is an arbitrarily fixed time, $\varepsilon>0$ is a sufficiently small constant, and $v$ is an arbitrary $U$-valued $\mathcal{F}_{\tau}$-measurable random variable such that $\mathbb{E}|v|^{3}<+\infty$. Let $x^{\varepsilon}(\cdot)$ be the trajectory of the control system (1) corresponding to the control $u^{\varepsilon}(\cdot)$.

The cost functional is

[TABLE]

where $l:[0,T]\times\mathbb{R}^{L}\times\mathbb{R}^{k}\times I\mathbb{\rightarrow R}$, $h:\mathbb{R}^{L}\mathbb{\rightarrow R}$. Our problem is to find an optimal control $u(\cdot)\in\mathcal{U}_{ad}$ such that

[TABLE]

In what follows, we add the following additional assumption:

(A3)

The map $l$ and $h$ are twice differentiable with the derivatives up to the second order being uniformly bounded and uniformly Lipschitz continuous in $u\in U$, $e_{i}\in S$.

We shall derive the variational inequality from the following fact:

[TABLE]

Next, fixed $\alpha(\cdot)$, for $\varphi=b,\sigma,\gamma$, we define

[TABLE]

Let $x_{1}(\cdot)$ and $x_{2}(\cdot)$ be respectively solutions to the following stochastic differential equations:

[TABLE]

and

[TABLE]

where

[TABLE]

and

[TABLE]

Remark 5**.**

Equations (8) and (9) are usually called the first and second-order variational equations, respectively.

Define the Hamiltonian function as follows:

[TABLE]

where $(t,x,u,e_{i},p,q,s)\in[0,T]\times\mathbb{R}^{L}\times U\times\mathbb{R}^{L}\mathbb{\times R}^{L\times d}\mathbb{\times R}^{L\times D}$.

The following result proceeds the Taylor expansion of the state with respect to the control perturbation. We need the following estimations:

Lemma 6**.**

Let (A1)-(A2) hold. Then, we have

[TABLE]

Proof.

Let $\xi^{\varepsilon}(t)\triangleq x^{\varepsilon}(t)-\bar{x}(t)$, $t\in[0,T]$. Then, we have

[TABLE]

where

[TABLE]

By the classical method, it is easy to verify $\sup\limits_{0\leq t\leq T}\mathbb{E}[|x_{1}(t)|^{2p}]=O\left(\varepsilon^{p}\right)$ and $\sup\limits_{0\leq t\leq T}\mathbb{E}[|x_{2}(t)|^{2p}]=O\left(\varepsilon^{2p}\right)$ for $p\geq 1.$ Taking the expectation after squaring both sides of (18) and using Gronwall inequality, we have

[TABLE]

Similarly, we can prove (14). We now deal with (15). In fact,

[TABLE]

To prove (16), let

[TABLE]

Then, it follows from (18) and (8) that we obtain

[TABLE]

Therefore,

[TABLE]

We consider the following term

[TABLE]

Similarly, we have

[TABLE]

and

[TABLE]

Hence, we get (16). Set

[TABLE]

Then, we have

[TABLE]

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

One can check that

[TABLE]

Recall

[TABLE]

From (19), we have

[TABLE]

It is easy to check that

[TABLE]

By virtue of Gronwall inequality, Itô’s formula and (20), we complete the proof. $\Box$

Lemma 7**.**

Let (A1)-(A3) hold. Then

[TABLE]

Proof. From the fact that $(\bar{x}(\cdot),\bar{u}(\cdot),\alpha(\cdot))$ is optimal triple, we have

[TABLE]

By Lemma 6, we get

[TABLE]

By Lemma 6, we complete the proof. $\Box$

Next, we shall find the first and second-order adjoint equations. Using these processes, we are able to derive the variational inequality from(21).

Consider the following stochastic system:

[TABLE]

where $(\phi(\cdot),\psi(\cdot),\mu(\cdot))\in\mathcal{M}^{2}(0,T;\mathbb{R}^{L})\times\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times d})\times\mathcal{V}^{2}(0,T;\mathbb{R}^{L\times D})$. We construct a linear functional on the Hilbert space $\mathcal{M}^{2}(0,T;\mathbb{R}^{L})\times\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times d})\times\mathcal{V}^{2}(0,T;\mathbb{R}^{L\times D})$ as follows:

[TABLE]

where $\phi(\cdot),\psi(\cdot),\mu(\cdot)$ and $z(\cdot)$ are defined in(22). Then, by virtue of the Riesz Representation Theorem, there exists a unique solution

[TABLE]

such that

[TABLE]

for $\forall(\phi(\cdot),\psi(\cdot),\mu(\cdot))\in\mathcal{M}^{2}(0,T;\mathbb{R}^{L})\times\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times d})\times\mathcal{V}^{2}(0,T;\mathbb{R}^{L\times D})$. We now consider equations (8) and (9) as the solution to (22), respectively. Then, we have

[TABLE]

and

[TABLE]

Then, using Assumption (A1) and Lemma 6, we can re-write the expression (21) as

[TABLE]

Next, we focus on the quadratic terms in (23) by employing the Riesz Representation Theorem. Applying Itô’s formula to $X(t)=x_{1}(t)x_{1}^{\top}(t)$ on $[0,T]$ yields

[TABLE]

Consider the following symmetric matrix-valued linear stochastic differential equations:

[TABLE]

where $(\Upsilon(\cdot),\Psi(\cdot),\Pi(\cdot))\in\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times L})\times(\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times L}))^{d}\times(\mathcal{V}^{2}(0,T;\mathbb{R}^{L\times L}))^{D}$.

Define a linear functional based on (25):

[TABLE]

Obviously, $\mathcal{I}_{2}(\cdot,\cdot,\cdot)$ is a linear continuous functional on $\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times L})\times(\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times L}))^{d}$ $\times(\mathcal{V}^{2}(0,T;\mathbb{R}^{L\times L}))^{D}$. There exists a unique solution $(P(\cdot),Q(\cdot),S(\cdot))\in\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times L})\times(\mathcal{M}^{2}(0,T;\mathbb{R}^{L\times L}))^{d}\times(\mathcal{V}^{2}(0,T;\mathbb{R}^{L\times L}))^{D}$, such that

[TABLE]

Now let

[TABLE]

Then, the relation (23) can be expressed as

[TABLE]

From the definition of $(\Upsilon^{\varepsilon}(\cdot),\Psi^{\varepsilon j}(\cdot),\Pi^{\varepsilon j}(\cdot))$ and Lemma 6, we have

[TABLE]

where

[TABLE]

Moreover, we adopt the scalar product: $\langle A,B\rangle=$tr$[AB]$, $A$, $B\in\mathbb{R}^{n\times n}$. It therefore follows that for all $y\in\mathbb{R}^{n}$ and $A\in\mathbb{R}^{n\times n}$, we have $\langle yy^{\top},A\rangle=$tr$[(yy^{\top})A]=y^{\top}Ay$.

Remark 8**.**

The inequality (28) is called the variational inequality of our optimal control problem. In contrast to the classical work by Peng [37], the appearance of extra term with $\lambda_{j}(t)$ is due to the random environments.

Next we shall find the second-order adjoint equation. The first one has been given in Zhang et al [45] like:

[TABLE]

From (26) and (28), we can find the second-order adjoint equation is

[TABLE]

where $H$ is defined in (12). Now define an $\mathcal{H}$-function:

[TABLE]

Inequality (28) can be expressed as

[TABLE]

Theorem 9**.**

Suppose that Assumptions (A1)-(A3) are in force. If $(\bar{x}(\cdot),\bar{u}(\cdot))$ is a solution to optimal control problem(7). Then

[TABLE]

which are, respectively, unique solutions to equations (29)-(30) such that the variational inequality (28) holds.

Proof. From (28) and discussion above, we complete the proof. $\Box$

We now establish the relationship between MP and DPP. To this end, we consider the Markovian (feedback) control, namely, the control $u(t)$ of the form $u(t,X(t-),\alpha(t-))$.

Define

[TABLE]

Let us define the value function as follows:

[TABLE]

for every $(t,x,e_{i})\in[0,T]\times\mathbb{R}^{L}\times S$.

Employing the classical dynamic programming principle (see Fleming and Soner [12]), we are able to attain the following Hamilton-Jacobi-Bellman (HJB, for short) equation with terminal boundary condition:

[TABLE]

where

[TABLE]

Put

[TABLE]

for $(t,x,p,P,e_{i})\in[0,T]\times\mathbb{R}^{L}\times U\times\mathbb{R}^{L}\times\mathbb{R}^{L\times L}\times S$.

Remark 10**.**

Clearly, because of the presence of regime-switching, (33) is a coupled system of nonlinear differential equations. Meanwhile, it is impossible for us to obtain the explicit form of the value function and an optimal control by solving (33). The main reason comes from the fact that (33) is a coupled system of nonlinear second-order partial differential equations, adding extreme difficulty in finding a closed-form solution of (33). Recall in lieu of a single nonlinear differential equation in the traditional literature for optimal control problems, we need more powerful tools and analysis to deal with it.

As a matter of fact, one should seek an optimal control and associated state trajectories. Normally, the main motivation for studying the dynamic principle programming is that one might construct an optimal control through the value function. The following result gives a way of testing whether a given admissible pair is optimal and, more importantly, suggests how to construct an optimal feedback control. Such a result is called a verification theorem. For smooth case, we have a following result.

Theorem 11** (Verification theorem).**

Assume that (A1)-(A3) hold. Let $V\in C^{1,2}([0,T]\times\mathbb{R}^{n}\times e_{i})$ for each $e_{i}\in S$ be a solution of HJB equation (33). Then

[TABLE]

Furthermore, an admissible pair $(\bar{x}(\cdot),\bar{u}(\cdot))$ is an optimal control for(32) if and only if

[TABLE]

Proof. Fix $e_{i}\in S$. Let $u(\cdot)\in\mathcal{U}_{ad}[t,T]$ with its associated state trajectory $x(\cdot)$. Applying Itô’ formula to $V(t,x,e_{i})$ on $[t,T]$, we have

[TABLE]

which implies (34). Applying (36) to $(\bar{x}(\cdot),\bar{u}(\cdot))$, together with (35), yields the desired result. $\Box$

Remark 12**.**

As claimed in Theorem 11, the value function requires to be smooth enough. As we have well known, the value function 32 is not necessarily smooth. Thus, one perhaps introduce the viscosity solution to study our control problems. As this complete remark of the existence is much longer than the present paper, it will be reported elsewhere.

Let us now provide a substantial example to demonstrate how to construct an optimal control by Theorem 9.

Example 13**.**

Consider the following control system $(L=d=1)$:

[TABLE]

where $\sigma:\mathbb{R}^{D}\rightarrow\mathbb{R}$, $\gamma:\mathbb{R}^{D}\rightarrow\mathbb{R}^{D}$ with the control domain being $[0,1]$ and the cost functional being

[TABLE]

Let $(\bar{x}(\cdot),\bar{u}(\cdot))$ be optimal pair to be determined. The first and second-order adjoint equations are

[TABLE]

and

[TABLE]

According to the existence and uniqueness of BSDE (40), the unique adapted the solution is $(P(t),Q(t),S(t))=(1,0,0)$, $t\in[0,T]$. The corresponding $\mathcal{H}$-function is

[TABLE]

The function $\mathcal{H}(s,\bar{x}(s),\cdot,e_{i})$ attains its minimum at

[TABLE]

Particularly, set $\sigma(e_{i})=1$, $\gamma(e_{i})\equiv 0$. Then

[TABLE]

Hence the necessary condition of optimality as specified by the stochastic maximum principle (Theorem 9) would be satisfied if one could find a control $u(\cdot)$ such that the corresponding $q(s)=1$, $\varsigma(s)=0$. By the first-order adjoint equation (39), it is clear that if we take $u(s)=1$ with the corresponding state $\bar{x}(s)=W(s)$, then the unique solution of (39) is $(p(s),q(s),\varsigma(s))=(W(s),1,0)$.

4 Optimal control for recursive utilities

Based on the results obtained in Section 3, we study the optimal control problem for stochastic recursive utilities for systems composed of SDE (1) and BSDE (2). The cost functional $J(u(\cdot))$ now is defined in (3). The control problem is to minimize $J(u(\cdot))$ over $\mathcal{U}_{ad}$. For simplicity, we assume that $L=1$, $d=1$. Let $\bar{u}(\cdot)$ be the optimal control and let $(\bar{x}(\cdot),\bar{y}(\cdot),\bar{z}(\cdot),\bar{\kappa}(\cdot))$ be the corresponding solution of the equations (1) and (2). Similarly, we define $(x^{\varepsilon}(\cdot),y^{\varepsilon}(\cdot),z^{\varepsilon}(\cdot),\kappa^{\varepsilon}(\cdot))$ for $u^{\varepsilon}(\cdot)$.

We shall give the variational equation for BSDE (2). For this aim, we consider the following two adjoint equations:

[TABLE]

and

[TABLE]

where $F(t)$ and $G(t)$ are adapted processes with suitable properties, will be determined later.

Applying Itô’s formula to $\langle\mathfrak{p}(t),x_{1}(t)+x_{2}(t)\rangle+\frac{1}{2}x_{2}(t)^{\top}\mathcal{P}(t)x_{2}(t)$ yields

[TABLE]

where

[TABLE]

From Lemma 6, we derive that

[TABLE]

and

[TABLE]

Let

[TABLE]

Then, a simple calculation yields

[TABLE]

Now let

[TABLE]

It is easy to show that

[TABLE]

Next, we deal with

[TABLE]

where

[TABLE]

We shall construct a linear BSDE with solution $(\hat{y}^{\varepsilon}(t),\hat{z}^{\varepsilon}(t),\hat{\kappa}^{\varepsilon}(t))$ via selecting certain suitable $F(t)$, $G(t)$. The following conditions must be satisfied:

• •

$F(t)$ and $G(t)$ are determined by $(s,\bar{x}(s),\bar{y}(s),\bar{z}(s),\bar{\kappa}(s),\bar{u}(s),\alpha(s-))$;

• •

We must have

[TABLE]

where $o(\varepsilon)$ is independent on $x_{1}(s)$ and $x_{2}(s)$.

Applying Taylor’s expansion to

[TABLE]

we are able to find

[TABLE]

where $f(t)=f(s,\bar{x}(s),\bar{y}(s),\bar{z}(s),\bar{\kappa}(s),\bar{u}(s),\alpha(s-))$ and similarly for $f_{x}$, $f_{y}(\cdot)$, $f_{z}(\cdot)$, $f_{\kappa}(\cdot)$ and $D^{2}f(\cdot)$. By Assumptions (A1) and(A2), we get that the adjoint equations (41) and (42) have unique solutions $(\mathfrak{p}(\cdot)\mathfrak{,q}(\cdot)\mathfrak{,s}(\cdot))$ and $(\mathcal{P}(\cdot),\mathcal{Q}(\cdot),\mathcal{S}(\cdot))$, respectively, and

[TABLE]

We introduce the following BSDE:

[TABLE]

Next, we shall prove the following estimates:

Lemma 14**.**

Assume that (A1)-(A2)

[TABLE]

[TABLE]

[TABLE]

Proof.

We first prove (45). We reformulate BSDE (43) as follows:

[TABLE]

Clearly,

[TABLE]

First, we consider

[TABLE]

where

[TABLE]

for $l=y$, $z$, $\kappa$, respectively. Next, we deal with the following term

[TABLE]

where

[TABLE]

It therefore follows that

[TABLE]

where

[TABLE]

It follows from Lemma 4 and Lemma 6 that we obtain (45). By the same argument, we can easily get (46). Next we handle (47). To this end, we set

[TABLE]

Subtracting (44) from BSDE (48) yields

[TABLE]

Applying Lemma 4, we only need to prove that

[TABLE]

[TABLE]

and

[TABLE]

Obviously,

[TABLE]

One needs to check that

[TABLE]

and

[TABLE]

From (4)-(55), we prove (50). Due to $D^{2}f$ is bounded the definition of $\check{D}^{2}f(s)$, we have

[TABLE]

which yields

[TABLE]

It is fairly easy to get that

[TABLE]

One can check that

[TABLE]

and

[TABLE]

Therefore, (52) holds. Finally, from Lemma 4 we complete the proof. $\Box$

Now we are able to show the variational equations for BSDE (2):

[TABLE]

Similar to $x_{1}(\cdot)$ and $x_{2}(\cdot)$, we have

[TABLE]

and

[TABLE]

Now let us come back to discuss about the maximum principle for optimal control of FBSDEs (1)-(2). From the definition of performance functional (3), it follows

[TABLE]

Define the adjoint equation for (2) ($\lambda_{j}\neq 0$):

[TABLE]

Clearly, SDE (60) admits a unique strong solution. By virtue of Itô’s formula to $\hat{y}(t)\chi(t)$, we can obtain

[TABLE]

In order to state our main result, we define a function as follows:

[TABLE]

We now assert the following main result.

Theorem 15**.**

Assume that (A1)-(A2) hold. Let $\bar{u}(\cdot)$ be an optimal control and $(\bar{x}(\cdot),\bar{y}(\cdot)$, $\bar{z}(\cdot),\bar{\kappa}(\cdot))$ be the associated solution. Then

[TABLE]

where $\mathbb{H}$ is defined in (61).

We also provide a concrete example.

Example 16**.**

Consider the following FBSDEs with $d=1$, $L=1$

[TABLE]

where $A\in\mathbb{R}^{D\times D}$ and $\nu_{j}(t)$ is a $\mathbb{R}$-valued deterministic process, $1\leq j\leq D$. Clearly, the solutions to equations (41) and (42) are $(\mathfrak{p}(t)\mathfrak{,q(t),s}(t))=(1,0,0)$, $(\mathcal{P}(t),\mathcal{Q}(t),\mathcal{S}(t))=(0,0,0)$, respectively. Therefore, from Theorem 15, the necessary condition for optimal control is

[TABLE]

For instance, let $f(z,\kappa)=z+\sum_{j=1}^{D}|\kappa_{j}|^{2}\lambda_{j}$. Then, we immediately derive that $\bar{u}\equiv 0$ is an optimal control. The corresponding trajectories are $(0,0,0,0)$.

5 Concluding remarks

In this paper, we study a general stochastic maximum principle for optimal control for systems governed by a continuous-time Markov regime-switching stochastic recursive utility. The control domain is assumed not to be convex, and the diffusion terms allow to contain control variables. On the one hand, we derive a result for forward stochastic optimal control problems. Afterwards, based on previous estimates and inspirations, we introduce two groups of new first and second-order adjoint equations. Moreover, the corresponding variational equations for forward-backward stochastic differential equations are obtained. Finally, we present our results using an illustrated example. It is necessary to point out that, due to the Markov regime-switching, the generator in the maximum principle involves solutions of the novel second-order adjoint equations. However, a number of other issues deserve further investigations as follows:

• •

Apart from Pontryagin’s maximum principle, the Bellman’s dynamic programming principle (DPP for short) is another important tool in tackling stochastic optimal control problems. Nowadays, many researchers engage in this field and achieve fundamental result. As for the DPP, systematic investigations of the classical stochastic optimal control problems are discussed deeply in the famous book by Fleming and Soner [12]. Now return to the expression of HJB equation (33). It vividly indicates that it is a coupled system of nonlinear differential equations because of the presence of regime-switching, in contrast to a single nonlinear differential equation in the traditional literature such as the user’s guide by Crandall et al [3]. Out of question, it is conceivable that the fruitful analysis will be much more involved than that in [3]. Nevertheless, we are not able to provide the explicit form of the value function and an optimal control via solving (33). The main obstacle appears since it is a coupled system of nonlinear second-order partial differential equations, rendering extreme difficulty in constructing a closed-form solution of (33). In future, we shall employ the viscosity solution to study this problem(see [39, 46] in this direction).

• •

An interesting topic is to consider the fully coupled FBSDEs. Indeed, some financial optimization problems for large investors and some asset pricing problems with forward-backward differential utility (see [5, 6, 30] directly lead to fully coupled FBSDEs. Using the ideas from Peng and Wu [38], under certain $G$-monotonicity conditions for the coefficients, we may derive the existence and uniqueness of fully coupled FBSDEs with Markov regime-switching. And then, a series of questions turn out, for instance, the associated partial differential equations and control problems. Another problem of great interests is to consider the case when the random environment or the Markov regime-switching $\alpha$ is unobservable.

Appendix A The Proof of Lemma 4

Proof of Lemma 4.

We first assume that $F$ are bounded. Let $\epsilon>0$ and define

[TABLE]

Clearly for any $\epsilon>0$, the function $y\rightarrow\langle y\rangle_{\epsilon}$ is smooth and $\langle y\rangle_{\epsilon}\rightarrow|y|$ as $\epsilon\rightarrow 0$. The motivation of introducing such a function is to avoid some difficulties that might be encountered in differentiating functions like $|y|^{2k}$ for non-integer $k$. Applying Itô’s formula to $\langle y\rangle_{\epsilon}^{2k}$ on $[t,T]$, we have

[TABLE]

Here $K_{0}=K_{0}(K,k)$ is independent of $t$. Applying the well-known Young’s inequality ($ab\leq\frac{a^{p}}{p}+\frac{b^{q}}{q}$ with $\frac{1}{p}+\frac{1}{q}=1$), we get

[TABLE]

Let $\beta_{1}$ be large enough such that $K_{1}=k(2k-1)-\frac{K_{0}}{\beta_{1}}>0$. It immediately yields

[TABLE]

Similarly, let $\beta_{2}$ be large enough such that $\tilde{K}_{1}=k(2k-1)-\frac{K_{0}}{\beta_{2}}>0$. Then, we have

[TABLE]

Hence, it follows from Gronwall’s inequality that

[TABLE]

and

[TABLE]

Note that $K_{2}=K_{2}(K,T,k)$. Besides since we postulate that at the beginning being that $F$ are bounded, the above procedure becomes valid (otherwise the integration on the right-hand side of (63) may not exist; see (5). Next, we want to refine the above estimate so that(6) will follow. To this end, observe that (65) implies that its left-hand side is bounded uniformly in $t\in[0,T]$. Thus, it is allows us to define

[TABLE]

We now come back (64), using (67). Define $\delta=\frac{1}{2K_{0}}-(1-\frac{1}{2k})\tau$, for $\tau$ small enough. Then, for any $t\in[T-\delta,T]$, we have from Young’s inequality, Hölder inequality and (66)

[TABLE]

The constant $K_{4}=K_{4}(k,K,\delta)$ in (68) is independent of $t$. Hence, it follows from (68) that

[TABLE]

Now we repeat the same procedure on $[\delta,2\delta]$ and on $[2\delta,3\delta]$, and so on. Eventually, we end up with

[TABLE]

with $K_{5}=K_{5}(k,\delta,K)$. According to the definition of (67), we conclude that

[TABLE]

Letting $\epsilon\rightarrow 0$, we get the desired result. In the case that we only have(5), we may use the usual approach of approximation. We thus complete the proof. $\Box$

**Acknowledgements. **The author wish to thank Dr. Shuai Jing and Dr. Zhongyang Sun for his conversation and suggestions.