On risk-sensitive piecewise deterministic Markov decision processes

TL;DR

This paper studies risk-sensitive control in piecewise deterministic Markov decision processes, establishing optimality equations, value iteration, and existence of optimal policies, improving results for continuous-time risk-sensitive decision models.

Contribution

It introduces a framework for risk-sensitive control in PDMDPs, proving optimality equations and policy existence under natural conditions, with significant improvements over prior results.

Findings

Established the optimality equation for the model.

Justified the value iteration algorithm.

Proved existence of deterministic stationary optimal policies.

Abstract

We consider a piecewise deterministic Markov decision process, where the expected exponential utility of total (nonnegative) cost is to be minimized. The cost rate, transition rate and post-jump distributions are under control. The state space is Borel, and the transition and cost rates are locally integrable along the drift. Under natural conditions, we establish the optimality equation, justify the value iteration algorithm, and show the existence of a deterministic stationary optimal policy. Applied to special cases, the obtained results already significantly improve some existing results in the literature on finite horizon and infinite horizon discounted risk-sensitive continuous-time Markov decision processes.

Equations

$$\frac{0}{0}:=0,0\cdot \infty :=0,\frac{1}{0}:=+\infty ,\infty -\infty :=\infty .$$

$$\tilde{q}({\Gamma }_S|x,a):=q({\Gamma }_S\setminus \{x\}|x,a)\ge 0$$

$$q(S|x,a)=0,{\bar{q}}_x=\underset{a\in A}{\sup }{q}_x(a)<\infty ,$$

$$\varphi (x,t+s)=\varphi (\varphi (x,t),s),\forall s,t\ge 0;\varphi (x,0)=x,$$

$$t_n(\omega )$$

$$t_{\infty }(\omega ):=\underset{n\to \infty }{\lim }{t}_n(\omega ).$$

$${\xi }_t=\sum _{n\ge 0}I\{t_n\le t<t_{n+1}\}\varphi (x_n,t-t_n)+I\{t_{\infty }\le t\}{x}_{\infty },t\ge 0,$$

$$\pi (da|\omega ,t)$$

$$P_{\gamma }^{\pi }({\theta }_{n+1}\in {\Gamma }_1,x_{n+1}\in {\Gamma }_2|x_0,{\theta }_1,x_1,\dots ,{\theta }_n,x_n)$$

$$P_{\gamma }^{\pi }({\theta }_{n+1}=\infty ,x_{n+1}=x_{\infty }|x_0,{\theta }_1,x_1,\dots ,{\theta }_n,x_n)=1.$$

$$E_x^{\pi }\left[e^{{\int }_0^{\infty }{\int }_{A}c({\xi }_t,a)\pi (da|\omega ,t)dt}\right]=E_x^{\pi }\left[e^{{\sum }_{n=0}^{\infty }{\int }_0^{{\theta }_{n+1}}{\int }_{A}c(\varphi (x_n,s),a){\pi }_n(da|x_0,{\theta }_1,\dots ,x_n,s)ds}\right]=:V(x,\pi )$$

$$\pi \in \Pi$$

$$-(V(\varphi (x,t))-V(x))$$

$$\underset{a\in A}{\inf }\left\{{\int }_S{V}^{\ast }(y)\tilde{q}(dy|x,a)-(q_x(a)-c(x,a))V^{\ast }(x))\right\}$$

$$-(V(\varphi (x,t))-V(x))$$

$$\underset{a\in A}{\inf }\left\{{\int }_S{V}^{\ast }(y)\tilde{q}(dy|x,a)-(q_x(a)-c(x,a))V^{\ast }(x))\right\}$$

$$-(V^{(n+1)}(\varphi (x,t))-V^{(n+1)}(x))$$

$$\displaystyle\mbox{Minimize over $\pi\in\Pi$: }E_{x}^{\pi}\left[e^{\int_{0}^{\infty}e^{-\alpha t}\int_{A}c(\xi_{t},a)\pi(da|\omega,t)dt}\right],~{}x\in S.$$

$$\underset{t\in [0,\infty )}{\sup }\{{\overline{q}}_{(t,x)}\}<\infty ,\forall x\in S,\underset{t\in [0,\infty ),x\in S,a\in A}{\sup }c(t,x,a)<\infty ,$$

$$-(V(t,x)-V(0,x))$$

$$\underset{a\in A}{\inf }\left\{{\int }_{S}L(u,y)\tilde{q}(dy|u,x,a)+(e^{-\alpha u}c(u,x,a)-q_{(u,x)}(a))L(u,x)\right\}$$

$$\hat{q}(ds\times dy|(t,x),a):=\tilde{\hat{q}}(ds\times dy|(t,x),a)-I\{(t,x)\in ds\times dy\}{q}_{(t,x)}(a),$$

$$\tilde{\hat{q}}(ds\times dy|(t,x),a):=I\{t\in ds\}\tilde{q}(dy|t,x,a),$$

$$\hat{c}((t,x),a):=e^{-\alpha t}c(t,x,a),\forall t\in [0,\infty ),x\in S,a\in A.$$

$${\hat{P}}_{\hat{\gamma }}^{\hat{\pi }}({\hat{\theta }}_{n+1}\in {\Gamma }_1,{\hat{x}}_{n+1}\in {\Gamma }_2\times {\Gamma }_3|{\hat{x}}_0,{\hat{\theta }}_1,{\hat{x}}_1,\dots ,{\hat{\theta }}_n,{\hat{x}}_n)$$

$$\displaystyle\mbox{Minimize over $\pi\in\Pi$: }E_{x}^{\pi}\left[e^{\int_{0}^{T}e^{-\alpha t}\int_{A}c(t,\xi_{t},a)\pi(da|\omega,t)dt+g(\xi_{T})}\right],~{}x\in S,$$

$$\displaystyle\hat{c}((t,x),a)=\left\{\begin{array}[]{ll}e^{-\alpha t}c(t,x,a),&\mbox{ if }t\leq T;\\ e^{-(t-T)}g(x)&\mbox{ if }t>T.\end{array}\right.$$

$$q_x(\mu ):={\int }_A{q}_x(a)\mu (da),\tilde{q}(dy|x,\mu ):={\int }_A\tilde{q}(dy|x,a)\mu (da),c(x,\mu ):={\int }_{A}c(x,a)\mu (da).$$

$$V^{\ast }(x)$$

Full text

\marginsize

2.0cm2.0cm2.0 cm2.0 cm

On risk-sensitive piecewise deterministic Markov decision processes

Xin Guo   and Yi Zhang Department of Mathematical Sciences, University of Liverpool, Liverpool, L69 7ZL, U.K.. E-mail: X.Guo21@liv.ac.uk.Corresponding author. Department of Mathematical Sciences, University of Liverpool, Liverpool, L69 7ZL, U.K.. E-mail: yi.zhang@liv.ac.uk.

Abstract: We consider a piecewise deterministic Markov decision process, where the expected exponential utility of total (nonnegative) cost is to be minimized. The cost rate, transition rate and post-jump distributions are under control. The state space is Borel, and the transition and cost rates are locally integrable along the drift. Under natural conditions, we establish the optimality equation, justify the value iteration algorithm, and show the existence of a deterministic stationary optimal policy. Applied to special cases, the obtained results already significantly improve some existing results in the literature on finite horizon and infinite horizon discounted risk-sensitive continuous-time Markov decision processes.

Keywords: Continuous-time Markov decision processes. Piecewise deterministic Markov decision processes. Exponential utility. Dynamic programming.

AMS 2000 subject classification: Primary 90C40, Secondary 60J75

1 Introduction

Since the pioneering work [18], risk-sensitive discrete-time Markov decision processes (DTMDPs) have been studied intensively. Having restricted our attention to total undiscounted or discounted problems, let us mention e.g., [3, 6, 7, 11, 12, 15, 16], most of which deal with the exponential utility, as well as in the present paper. As an application, an open problem in insurance was recently solved in [4] in the framework of risk-sensitive DTMDP. There are notable differences between risk-sensitive and risk-neutral DTMDPs. For instance, in a finite model, i.e., when the state and action spaces are both finite, there is always a deterministic stationary optimal policy in a discounted risk-neutral DTMDP, but not always in a discounted risk-sensitive DTMDP, see [15].

One of the first works on risk-sensitive continuous-time Markov decision processes (CTMDPs) is [21], where only verification theorems were presented. Recently, there have been reviving interests in this topic; see e.g., [8, 14, 20, 24, 25, 27]. A finite horizon total undiscounted risk-sensitive CTMDP was considered in [14, 21, 24], whose arguments were summarized as follows. Firstly, the optimality equation is shown to admit a solution out of a small enough class. Secondly, by using the Feynman-Kac formula, this solution is shown to be the value function, and any Markov policy providing the minimizer in the optimality equation is optimal. The proofs of [14, 24] reveal that the the main technicalities lie in the first step, for which, the state space was assumed to be denumerable. This assumption is important for the diagonalization argument used in [24], which is an extension of [14] from bounded transition rate to possibly unbounded transition rate, whose growth is bounded by a Lyapunov function. The latter requirement and the boundedness of the cost rate then validate the Feynman-Kac formula applied in the second step. The author of [24] mentioned that it was unclear how to extend his argument to an unbounded cost rate, see Section 7 therein. Following a similar argument as described above, a discounted risk-sensitive CTMDP was also considered in [14], although now the first step becomes, to quote the authors’ words (see p.658 therein), “surprisingly far more involved”, for which the state space was further assumed to be finite, see Remark 3.6 therein. It is a corollary of the present paper that we significantly weaken the restrictive conditions in [14, 24], see Section 3 below.

The present paper is concerned with a risk-sensitive piecewise deterministic Markov decision process (PDMDP), where the expected exponential utility of the total cost is to be minimized. The state space is a general Borel space, the transition and the nonnegative cost rates only need be locally integrable along the drift. A PDMDP is an extension of a CTMDP: now between two consecutive jumps, the process evolves according to deterministic Markov process. For simplicity and to keep the conditions as weak as possible, we do not consider the control on the drift. In spite that there has been a vast literature on PDMDPs; see the well known monographs [9, 10] and the references therein, to the best of our knowledge, risk-sensitive PDMDPs have not been systematically studied before.

Our main contributions are the following. We establish the optimality equation satisfied by the value function, justify the value iteration algorithm and show the existence of a deterministic stationary optimal policy. As an application and corollary, finite horizon and infinite horizon discounted risk-sensitive CTMDPs are reformulated as total undiscounted risk-sensitive PDMDPs, and are thus treated in a unified way and under much weaker conditions than in [14, 24]. This is possible because we follow a different argument. Namely, we directly show that the value function satisfies the optimality equation, by reducing the total undiscounted risk-sensitive PDMDP to a risk-sensitive DTMDP. This method, without referring to the Feynman-Kac formula, was originally developed by Yushkevich [26] for risk-neutral CTMDPs. Later, it was employed in [1, 2, 9, 10, 13, 23] for studies of risk-neutral PDMDPs, and in [27] for risk-sensitive CTMDPs. In [8], restricted to stationary policies, the discounted risk-sensitive CTMDP with bounded transition rates was reduced to a DTMDP problem, using the uniformization technique. The induced DTMDP is less standard (with a random cost), and was not further investigated there.

The rest of the paper is organized as follows. In Section 2 we describe the concerned optimal control problem. In Section 3 we present the main results, the proofs of which are postponed to Section 4. We finish the paper with a conclusion in Section 5. Some relevant facts were collected in the appendix for ease of reference.

2 Model description and problem statement

Notations and conventions. In what follows, ${\cal{B}}(X)$ is the Borel $\sigma$-algebra of the topological space $X,$ $I$ stands for the indicator function, and $\delta_{\{x\}}(\cdot)$ is the Dirac measure concentrated on the singleton $\{x\},$ assumed to be measurable. A measure is $\sigma$-additive and $[0,\infty]$-valued. Below, unless stated otherwise, the term of measurability is always understood in the Borel sense. Throughout this paper, we adopt the conventions of

[TABLE]

If a mapping $f$ defined on $X$, and $\{X_{i}\}$ is a partition of $X$, then when $f$ is piecewise defined as $f(x)=g_{i}(x)$ for all $x\in X_{i}$, the notation $f(x)=\sum_{i}I\{x\in X_{i}\}g_{i}(x)$ is used, even if $f$ is not real-valued.

Let $S$ be a nonempty Borel state space, $A$ be a nonempty Borel action space, and $q$ stand for a signed kernel $q(dy|x,a)$ on ${\cal{B}}(S)$ given $(x,a)\in S\times A$ such that

[TABLE]

for all $\Gamma_{S}\in{\cal{B}}(S).$ Throughout this article we assume that $q(\cdot|x,a)$ is conservative and stable, i.e.,

[TABLE]

where $q_{x}(a):=-q(\{x\}|x,a).$ The signed kernel $q$ is often called the transition rate. Between two consecutive jumps, the state of the process evolves according to a measurable mapping $\phi$ from $S\times[0,\infty)$ to $S$, see (5) below. It is assumed that for each $x\in S$

[TABLE]

and $t\rightarrow\phi(x,t)$ is continuous.

Finally let the cost rate $c$ be a $[0,\infty)$-valued measurable function on $S\times A$. For simplicity, we do not consider the case of different admissible action spaces at different states.

Condition 2.1

• (a)

For each bounded measurable function $f$ on $S$ and each $x\in S$, $\int_{S}f(y)\tilde{q}(dy|x,a)$ is continuous in $a\in A.$

• (b)

For each $x\in S,$ the (nonnegative) function $c(x,a)$ is lower semicontinuous in $a\in A.$

• (c)

The action space $A$ is a compact Borel space.

Condition 2.2

For each $x\in S$, $\int_{0}^{t}\overline{q}_{\phi(x,s)}ds<\infty$, and $\int_{0}^{t}\sup_{a\in A}c(\phi(x,s),a)ds<\infty$, for each $t\in[0,\infty).$

The integrals in the above condition are well defined: the integrands are universally measurable in $s\in[0,\infty)$; see Chapter 7 of [5].

Let us take the sample space $\Omega$ by adjoining to the countable product space $S\times((0,\infty)\times S)^{\infty}$ the sequences of the form $(x_{0},\theta_{1},\dots,\theta_{n},x_{n},\infty,x_{\infty},\infty,x_{\infty},\dots),$ where $x_{0},x_{1},\dots,x_{n}$ belong to $S$, $\theta_{1},\dots,\theta_{n}$ belong to $(0,\infty),$ and $x_{\infty}\notin S$ is the isolated point. We equip $\Omega$ with its Borel $\sigma$-algebra $\cal F$.

Let $t_{0}(\omega):=0=:\theta_{0},$ and for each $n\geq 0$, and each element $\omega:=(x_{0},\theta_{1},x_{1},\theta_{2},\dots)\in\Omega$, let

[TABLE]

and

[TABLE]

Obviously, $(t_{n}(\omega))$ are measurable mappings on $(\Omega,{\cal F})$. In what follows, we often omit the argument $\omega\in\Omega$ from the presentation for simplicity. Also, we regard $x_{n}$ and $\theta_{n+1}$ as the coordinate variables, and note that the pairs $\{t_{n},x_{n}\}$ form a marked point process with the internal history $\{{\cal F}_{t}\}_{t\geq 0},$ i.e., the filtration generated by $\{t_{n},x_{n}\}$; see Chapter 4 of [19] for greater details. The marked point process $\{t_{n},x_{n}\}$ defines the stochastic process $\{\xi_{t},t\geq 0\}$ on $(\Omega,{\cal F})$ of interest by

[TABLE]

where we accept $0\cdot x:=0$ and $1\cdot x:=x$ for each $x\in S_{\infty},$ and below we denote $S_{\infty}:=S\bigcup\{x_{\infty}\}$.

A (history-dependent) policy $\pi$ is given by a sequence $(\pi_{n})$ such that, for each $n=0,1,2,\dots,$ $\pi_{n}(da|x_{0},\theta_{1},\dots,x_{n},s)$ is a stochastic kernel on $A$, and for each $\omega=(x_{0},\theta_{1},x_{1},\theta_{2},\dots)\in\Omega$, $t>0,$

[TABLE]

where $a_{\infty}\notin A$ is some isolated point. A policy $\pi$ is called Markov if, with slight abuse of notations, $\pi(da|\omega,s)=\pi^{M}(da|\xi_{s-},s)$ for some stochastic kernel $\pi^{M}$. A Markov policy is further called deterministic if the stochastic kernels $\pi^{M}(da|x,s)=\delta_{\{f^{M}(x,s)\}}(da)$ for some measurable mapping $f^{M}$ from $S\times(0,\infty)$ to $A$. A policy is called deterministic stationary if for each $n=0,1,\dots,$ $\pi_{n}(da|x_{0},\theta_{1},\dots,\theta_{n},x_{n},t-t_{n})=\delta_{\{f(\phi(x_{n},t-t_{n}))\}}(da)$ for some measurable mapping $f$ from $S$ to $A$. We shall identify such a deterministic stationary policy by the underlying measurable mapping $f$.

The class of all policies is denoted by $\Pi.$ Under a fixed policy $\pi=(\pi_{n})$, for each initial distribution $\gamma$ on $(S,{\cal B}(S)),$ by using the Ionescu-Tulcea theorem, one can build a probability measure $P_{\gamma}^{\pi}$ on $(\Omega,{\cal F})$ such that $P_{\gamma}^{\pi}(x_{0}\in\Gamma)=\gamma(\Gamma)$ for each $\Gamma\in{\cal B}(S)$, and the conditional distribution of $(\theta_{n+1},x_{n+1})$ with the condition on $x_{0},\theta_{1},x_{1},\dots,\theta_{n},x_{n}$ is given on $\{\omega:x_{n}(\omega)\in S\}$ by

[TABLE]

and given on $\{\omega:x_{n}(\omega)=x_{\infty}\}$ by

[TABLE]

Below, when $\gamma$ is a Dirac measure concentrated at $x\in S,$ we use the denotation ${}{P}_{x}^{\pi}.$ Expectations with respect to ${}{P}_{\gamma}^{\pi}$ and ${}{P}_{x}^{\pi}$ are denoted as ${}{E}_{\gamma}^{\pi}$ and ${}{E}_{x}^{\pi},$ respectively. Roughly speaking, the uncontrolled version of the process evolves as follows: given the current state, the process evolves deterministically according to the mapping $\phi$, up to the next jump, taking place after a random time whose distribution is (nonstationary) exponential, and the dynamics continue in the similar manner. A detailed book treatment with many examples of this and more general type of processes, allowing deterministic jumps, can be found in [10].

For each $x\in S$, and policy $\pi=(\pi_{n})$,

[TABLE]

defines the concerned performance measure of the policy $\pi\in\Pi$ given the initial state $x\in S.$ Here and below, we put $c(x_{\infty},a):=0$ for each $a\in A,$ and $\phi(x_{\infty},t)=x_{\infty}$ for each $t\in[0,\infty).$ We are interested in the following optimal control problem for each $x\in S:$

[TABLE]

A policy $\pi^{\ast}$ is called optimal if $V(x,\pi^{\ast})=\inf_{\pi\in\Pi}V(x,\pi)=:V^{\ast}(x)$ for each $x\in S$.

The objective of this paper is to show, under the imposed conditions, the existence of a deterministic stationary optimal policy, and to establish the corresponding optimality equation satisfied by the value function $V^{\ast}$, together with its value iteration. Evidently, $V^{\ast}(x)\geq 1$ for each $x\in S.$ Under the next condition, it will be seen that for each $x\in S,$ $V^{\ast}(\phi(x,s))$ is absolutely continuous in $s.$

Condition 2.3

For each $x\in S,$ $V^{\ast}(x)<\infty$.

The above condition is mainly assumed for notational convenience. In fact, the main optimality results (such as the existence of a deterministic stationary optimal policy) obtained in this paper can be established without assuming Condition 2.3, at the cost of some additional notations. In a nutshell, one has to consider the sets $\hat{S}:=\{x\in S:~{}V^{\ast}(x)<\infty\}$ and $S\setminus\hat{S}$ separately, and note that if $x\in\hat{S}$, then $\phi(x,t)\in\hat{S}$ for each $t\in[0,\infty).$ The reasoning presented under Condition 2.3 can be followed in an obvious manner. We formulate the corresponding optimality results in Remarks 3.1 and 3.2 below.

3 Main statements

We first present the main optimality results concerning problem (8) for the PDMDP model. Their proofs are postponed to the next section.

Theorem 3.1

Suppose Conditions 2.1, 2.2 and 2.3 are satisfied. Then the following assertions hold.

• (a)

The value function $V^{\ast}$ for problem (8) is the minimal $[1,\infty)$-valued solution to the following optimality equation:

[TABLE]

In particular, $V^{\ast}(\phi(x,t))$ is absolutely continuous in $t$ for each $x\in S.$

• (b)

There exists a deterministic stationary optimal policy $f$, which can be taken as any measurable mapping from $S$ to $A$ such that

[TABLE]

Remark 3.1

By inspecting its proof, one can see the following version of Theorem 3.1 holds without assuming Condition 2.3. Suppose Conditions 2.1 and 2.2 are satisfied. Then the following assertions hold.

• (a)

The value function $V^{\ast}$ for problem (8) is the minimal $[1,\infty]$-valued solution to the following optimality equation:

[TABLE]

In particular, $V^{\ast}(\phi(x,t))$ is absolutely continuous in $t$ for each $x\in\hat{S}.$

• (b)

There exists a deterministic stationary optimal policy $f$, which can be taken as any measurable mapping from $S$ to $A$ such that

[TABLE]

Next, we present the value iteration algorithm for the value function $V^{\ast}$.

Theorem 3.2

Suppose Conditions 2.1, 2.2 and 2.3 are satisfied. Let $V^{(0)}(x):=1$ for each $x\in S$. For each $n\geq 0,$ let $V^{(n+1)}$ be the minimal $[1,\infty)$-valued measurable solution to

[TABLE]

such that $V^{(n+1)}(\phi(x,t))$ is absolutely continuous in $t$ for each $x\in S.$ (For each $n\geq 0,$ such a solution always exists.) Furthermore, $\{V^{(n)}\}$ is a monontone nondecreasing sequence of measurable functions on $S$ such that for each $x\in S,$ $V^{(n)}(x)\uparrow V^{\ast}(x)$ as $n\uparrow\infty.$

Remark 3.2

Similar to Remark 3.1, we have the following version of Theorem 3.2 without assuming Condition 2.3. Suppose Conditions 2.1, 2.2 are satisfied. Let $V^{(0)}(x):=1$ for each $x\in\hat{S}$ and $V^{(0)}(x)=\infty$ if $x\in S\setminus\hat{S}$. For each $n\geq 0,$ let $V^{(n+1)}$ be the minimal $[1,\infty]$-valued measurable solution to

[TABLE]

Here $V^{(n+1)}(\phi(x,t))$ is absolutely continuous in $t$ for each $x\in\hat{S}.$ (For each $n\geq 0,$ such a solution always exists.) Furthermore, $\{V^{(n)}\}$ is a monontone nondecreasing sequence of measurable functions on $S$ such that for each $x\in S,$ $V^{(n)}(x)\uparrow V^{\ast}(x)$ as $n\uparrow\infty.$

We can apply our theorems to a special case of a CTMDP. That is, $\phi(x,t)\equiv x$ for each $x\in S.$ The following $\alpha$-discounted risk-sensitive CTMDP problem was considered in [14]:

[TABLE]

Here $\alpha>0$ is a fixed constant. In fact, the authors of [14] were restricted to Markov policies, bounded transition and cost rates, i.e., $\sup_{x\in S}\overline{q}_{x}<\infty$, and $\sup_{x\in S,a\in A}c(x,a)<\infty$, and a finite state space $S$. These restrictions, e.g., the finiteness of $S$, were needed for their investigations, see e.g., Remark 3.6 in [14]. Under the compactness-continuity condition (Condition 2.1), it was shown in [14] that there exists an optimal Markov policy for the discounted risk-sensitive CTMDP, and established the optimality equation. By using the theorems presented earlier in this section, we can obtain these optimality results for problem (10) in a much more general setup: the state space $S$ is Borel, there is no boundedness requirement on the transition rate with respect to the state $x\in S$, and the optimality is over the class of history-dependent policies. Furthermore, we let the CTMDP model be nonhomogeneous, i.e., the transition rate $q(dy|t,x,a)$ now is a signed kernel on ${\cal B}(S)$ from $(t,x,a)\in[0,\infty)\times S\times A$, satisfying the corresponding version of (3); the notations $\tilde{q}$ is kept as before, see (2), with the extra argument $t$ in addition to $x$. Similarly, the nonnegative cost rate $c$ is allowed to be a measurable function on $[0,\infty)\times S\times A$.

Corollary 3.1

Consider the $\alpha$-discounted risk-sensitive (nonhomogeneous) CTMDP problem (10) with $c(\xi_{t},a)$ being replaced by $c(t,\xi_{t},a)$. Suppose

[TABLE]

and the corresponding version of Condition 2.1, where $x$ is replaced by $(t,x)$, is satisfied by the nonhomogeneous CTMDP model. Then the following assertions hold.

• (a)

There exists some $[1,\infty)$-valued measurable solution on $[0,\infty)\times S$ to

[TABLE]

so that $V(t,x)$ is absolutely continuous in $t$ for each $x\in S.$

• (b)

Let $L$ be the minimal $[1,\infty)$-valued measurable solution on $[0,\infty)\times S$ to the above equation. Then the value function say $L^{\ast}$ to the $\alpha$-discounted risk-sensitive CTMDP problem (10) (with $c(\xi_{t},a)$ being replaced by $c(t,\xi_{t},a)$) is given by $L^{\ast}(x)=L(0,x)$ for each $x\in S.$

• (c)

There exists an optimal deterministsic Markov policy $f$ for the $\alpha$-discounted risk-sensitive CTMDP problem (10) (with $c(\xi_{t},a)$ being replaced by $c(t,\xi_{t},a)$). One can take $f$ as any measurable mapping from $[0,\infty)\times S$ to $A$ such that

[TABLE]

for each $u\in[0,\infty)$ and $x\in S.$

Proof. We prove this by reformulating the nonhomogeneous version of the $\alpha$-discounted risk-sensitive (nonhomogeneous) CTMDP problem (10) in the form of problem (8) for a PDMDP, which we introduce as follows. We use the notation “hat” to distinguish this model from the original (nonhomogeneous) CTMDP model.

• •

The state space is $\hat{S}=[0,\infty)\times S.$

• •

The action space is the same as in the CTMDP: $\hat{A}=A.$

• •

the transition rate $\hat{q}(ds\times dy|(t,x),a)$ is defined by

[TABLE]

where

[TABLE]

for each $(t,x)\in\hat{S}$ and $a\in\hat{A}.$

• •

The drift is given by $\hat{\phi}((t,x),s):=(t+s,x)$ for each $x\in S$ and $t,s\geq 0.$ Clearly it satisfies the corresponding version of (4).

• •

The cost rate is given by

[TABLE]

Now the marked point process $\{\hat{t}_{n},\hat{x}_{n}\}$ and controlled process $\hat{\xi}_{t}$ in this PDMDP model is connected to those in the original (nonhomogeneous) CTMDP model, namely $(t_{n},x_{n})$ and $\xi_{t}$, via $\hat{t}_{n}=t_{n}$ and $\hat{x}_{n}=(t_{n},x_{n}),$ and $\hat{\xi}_{t}=(t,\xi_{t}).$ For example, under a fixed strategy $\hat{\pi}$ and initial distribution $\hat{\gamma}$ in this PDMDP model, the version of the first equation in (7) now reads on $\{\omega:x_{n}(\omega)\in S\}$

[TABLE]

Clearly, Conditions 2.1, 2.2 and 2.3 are satisfied by this PDMDP model. It remains to apply Theorem 3.1. $\hfill\Box$

The condition in the previous corollary is much weaker than in [14], and can be further weakened; one only needs the reformulated PDMDP to satisfy Conditions 2.1, 2.2 and 2.3. Moreover, the boundedness of the cost rate $c$ was assumed in the previous corollary only to ensure Condition 2.3 to be satisfied. It can be relaxed if one formulates the previous corollary using the statements in Remarks 3.1 and 3.2.

One can also consider the risk-sensitive nonhomogeneous CTMDP problem on the finite horizon $[0,T]$ with $T>0$ being a fixed constant:

[TABLE]

where $g$ is a $[0,\infty)$-valued measurable function; $g(x)$ represents the terminal cost incurred when $\xi_{T}=x\in S$. Let us put $g(x_{\infty}):=0.$ Here $\alpha$ is a fixed nonnegative finite constant. A simpler version of this problem was considered in [24] with $\alpha=0$ and a bounded cost rate, where additional restrictions were put on the growth of the transition rate. We can reformulate this problem into the PDMDP problem (8) just as in the above. The only difference is that now we put $q_{(t,x)}(a)\equiv 0$ for each $x\in S$ and $t\geq T,$ and introduce the following cost rate for each $x\in S$, $t\geq 0$ and $a\in A:$

[TABLE]

4 Proof of the main statements

For the rest of this paper, it is convenient to introduce the following notations. Let $\mathbb{P}(A)$ be the space of probability measures on ${\cal B}(A)$, endowed with the standard weak topology. For each $\mu\in\mathbb{P}(A)$,

[TABLE]

Let ${\cal R}$ denote the set of (Borel) measurable mappings $\rho_{t}(da)$ from $t\in(0,\infty)\rightarrow\mathbb{P}(A).$ Here, we do not distinguish two measurable mappings in $t\in(0,\infty),$ which coincide almost everywhere with respect to the Lebesgue measure. Let us equip ${\cal R}$ with the Young topology, which is the weakest topology with respect to which the function $\rho\in{\cal{R}}\rightarrow\int_{0}^{\infty}\int_{A}f(t,a)\rho_{t}(da)dt$ is continuous for each strongly integrable Carathéodory function $f$ on $(0,\infty)\times A$ . Here a real-valued measurable function $f$ on $(0,\infty)\times A$ is called a strongly integrable Carathéodory function if for each fixed $t\in(0,\infty)$, $f(t,a)$ is continuous in $a\in A,$ and for each fixed $a\in A,$ $\sup_{a\in A}|f(t,a)|$ is integrable in $t$, i.e., $\int_{0}^{\infty}\sup_{a\in A}|f(t,a)|dt<\infty.$ It is known that if $A$ is a compact Borel space, then so is ${\cal R}$; see Chapter 4 of [10].

Lemma 4.1

Suppose Conditions 2.1 and 2.2 are satisfied. Then the following assertions hold.

• (a)

The value function $V^{\ast}$ is the minimal $[1,\infty]$-valued measurable solution to

[TABLE]

• (b)

The mapping

[TABLE]

is lower semicontinuous for each $x\in S.$

Proof. One can legitimately consider the following DTMDP (discrete-time Markov decision process): according to Lemma 2.29 of [9], all the involved mappings are measurable.

• •

The state space is $\textbf{X}:=((0,\infty)\times S)\bigcup\{(\infty,x_{\infty})\}$. Whenever the topology is concerned, $(\infty,x_{\infty})$ is regarded as an isolated point in $\textbf{X}.$

• •

The action space is $\textbf{A}:={\cal R}$.

• •

The transition kernel $p$ on ${\cal B}(\textbf{X})$ from $\textbf{X}\times\textbf{A}$, c.f. (7), is given for each $\rho\in\textbf{A}$ by

[TABLE]

• •

The cost function $l$ is a $[0,\infty]$-valued measurable function on $\textbf{X}\times\textbf{A}\times\textbf{X}$ given by

[TABLE]

The relevant facts and statements for the DTMDP are included in the Appendix.

One can show that under Conditions 2.1 and 2.2, for each $(\theta,x)\in\textbf{X}$, $a\in\textbf{A}\rightarrow\int_{\textbf{X}}f(z)p(dz|(\theta,x),a)$ is continuous for each bounded measurable function $f$ on X; for each $(\theta,x)\in\textbf{X}$ and $(\tau,y)\in\textbf{X}$, $a\in\textbf{A}\rightarrow l((\theta,x),\rho,(\tau,y))$ is lower semicontinuous, and A is a compact Borel space. Hence, Condition A.1 for the DTMDP model $\{\textbf{X},\textbf{A},p,l\}$ is satisfied.

The controlled process in the above DTMDP model $\{\textbf{X},\textbf{A},p,l\}$ is denoted by $\{Y_{n},n=0,1,\dots\}$, where $Y_{n}=(\Theta_{n},X_{n})$, and the controlling process is denoted by $\{A_{n},n=0,1,\dots\}.$ For $n\geq 1,$ $\Theta_{n}$ and $X_{n}$ correspond to the $n$th sojourn time and the post-jump state in the PDMDP, $\Theta_{0}$ is fictitious, and $X_{0}$ is the initial state in the PDMDP. Let $\Sigma$ be the class of all strategies for the DTMDP model $\{\textbf{X},\textbf{A},p,l\}$, and $\Sigma_{DM}^{0}$ be the class of deterministic Markov strategies in the form $\sigma=(\varphi_{n})$ where $\varphi_{0}((\theta,x))$ does not depend on $\theta\in(0,\infty)$ for each $x\in S.$ We preserve the term of policy for the PDMDP and the term of strategy for the DTMDP.

According to Proposition A.1, the function

[TABLE]

is the minimal $[1,\infty]$-valued measurable solution to the optimality equation

[TABLE]

for each $x\in S$ and $\theta\in(0,\infty);$ this is just (22). Furthermore, by Proposition A.1, there exists a deterministic stationary strategy $\sigma^{\ast}$ for the DTMDP such that $\sigma^{\ast}((\theta,x))$ attains the above infimum for each $x\in S$ and $\theta\in(0,\infty),$ and any such strategy $\sigma^{\ast}$ verifies

[TABLE]

Let $\hat{\theta}\in~{}(0,\infty)$ be arbitrarily fixed. The function $\textbf{V}^{\ast}((\theta,x))$ being measurable in $(\theta,x)\in\textbf{X}$, it follows that $x\in S\rightarrow\textbf{V}^{\ast}((\hat{\theta},x))$ is measurable. The strategy $\sigma^{\ast}$ and the constant $\hat{\theta}$ induce a deterministic Markov strategy $\sigma^{\ast\ast}=(\varphi_{n})\in\Sigma^{0}_{DM}$, where $\varphi_{0}((\theta,x))=:\sigma^{\ast}((\hat{\theta},x))$ for each $\theta\in(0,\infty),~{}x\in S$, and $\varphi_{n}((\theta,x)):=\sigma((\theta,x))$ for each $n\geq 1$, $\theta\in(0,\infty),~{}x\in S.$ (The control on the isolated point $(0,x_{\infty})$ is irrelevant and we do not specify the definition of the strategy on that point.) This strategy can be identified with a policy $\pi^{\ast}$ in the PDMDP, c.f. (6). On the other hand, each policy $\pi=(\pi_{n})$ can be identified with a deterministic strategy in this DTMDP. Thus,

[TABLE]

for each $x\in S.$ Consequently, the policy $\pi^{\ast}$ is optimal, $V^{\ast}(x)=\textbf{V}^{\ast}((\theta,x))$ for each $x\in S$ and $\theta\in(0,\infty);$ recall that $\hat{\theta}$ was arbitrarily fixed. The statement of this lemma now follows. $\hfill\Box$

The policy $\pi^{\ast}$ in the proof of the previous lemma is actually optimal for problem (8). However, it is not necessarily a deterministic nor stationary policy. Also the reduction of the risk-sensitive PDMDP problem (8) to a risk-sensitive problem for the DTMDP model $\{\textbf{X},\textbf{A},p,l\}$ as seen in the proof of the above theorem will be used without special reference in what follows.

Lemma 4.2

Suppose Conditions 2.1, 2.2 and 2.3 are satisfied. For each $x\in S$ and $\rho\in{\cal R}$,

[TABLE]

is monotone nondecreasing in $t\in[0,\infty)$.

Proof. Let $0\leq t_{1}<t_{2}<\infty$ be arbitrarily fixed. We need show

[TABLE]

It is without loss of generality to assume

[TABLE]

Then all the four terms in (12) are nonnegative and finite, and (12) is equivalent to

[TABLE]

which is verified as follows. Let $\delta>0$ be arbitrarily fixed. By Lemma 4.1, there exists some $\hat{\nu}\in{\cal R}$ such that

[TABLE]

(Recall $\phi(x,t_{2}+t)=\phi(\phi(x,t_{2}),t)$ for each $t\geq 0.$) Consider $\tilde{\nu}\in{\cal R}$ defined by

[TABLE]

Then routine calculations lead to

[TABLE]

Since $\delta>0$ was arbitrarily fixed, now it follows that the term in the parenthesis in (13) is nonnegative, and thus inequality (13) is verified. $\hfill\Box$

Lemma 4.3

Suppose Conditions 2.1, 2.2 and 2.3 are satisfied. For each $x\in S$, there is some $\rho^{\ast}\in{\cal R}$ such that

[TABLE]

Proof. Let $x\in S$ be fixed, and let $\rho^{\ast}\in{\cal R}$ be such that $V^{\ast}(x)=W(x,\rho^{\ast})$, see Lemma 4.1. Suppose $t\in[0,\infty)$ is arbitrarily fixed. Consider $\tilde{\rho}\in{\cal R}$ defined by $\tilde{\rho}_{s}=\rho^{\ast}_{t+s}$ for each $s>0$. Then

[TABLE]

recall (4). On the other hand, by Lemma 4.2,

[TABLE]

The statement of this lemma is thus proved. $\hfill\Box$

Lemma 4.4

Suppose Conditions 2.1, 2.2 and 2.3 are satisfied. Then for each $x\in S,$ $t\in[0,\infty)\rightarrow V^{\ast}(\phi(x,t))$ is absolutely continuous.

Proof. This immediately follows from Lemma 4.3. $\hfill\Box$

Proof of Theorem 3.1. (a) Under Conditions 2.1, 2.2 and 2.3, by Lemma 4.4, for each $x\in S,$ let $t\in[0,\infty)\rightarrow U^{\ast}(x,t)$ be an integrable real-valued function such that $U^{\ast}(x,t)$ coincides with the derivative of $t\in[0,\infty)\rightarrow V(\phi(x,t))$ almost everywhere. Let $x\in S$ and $t\in[0,\infty)$ be fixed, and let $\rho^{\ast}\in{\cal R}$ be from Lemma 4.3.

By Lemmas 4.3 and 4.4,

[TABLE]

and

[TABLE]

are absolutely continuous in $\tau$ and are finite for each $\tau\in[0,\infty)$. Since $\phi(x,0)=x$, see (4),

[TABLE]

Now by Lemma 4.3,

[TABLE]

where $f$ is a measurable mapping from $S$ to $A$ such that

[TABLE]

for each $x\in S$; the existence of such a mapping is according to a well known measurable selection theorem, c.f. Proposition D.5 of [17].

Note that $e^{-\int_{0}^{\tau}(q_{\phi(x,v)}(\rho_{v})-c(\phi(x,v),\rho_{v}))dv}$ is bounded and separated from zero in $\tau\in[0,t]$ for each $\rho\in{\cal R};$ recall Condition 2.2. So

[TABLE]

is finite. If

[TABLE]

then

[TABLE]

which is against (16). Therefore,

[TABLE]

Then

[TABLE]

is absolutely continuous on $[0,t].$ After legitimately differentiating the above expression with respect to $v$, and applying Lemma 4.2, we see

[TABLE]

for almost all $v\in[0,t].$ This and (16) imply

[TABLE]

almost everywhere in $\tau\in[0,t].$ Remember, $t\in[0,\infty)$ was arbitrarily fixed. The first part of (a) is thus verified, and we postpone the justification of the second part of (a) after the proof of part (b).

(b) We use the same notation as in the above. Note that

[TABLE]

Indeed, if either $\int_{0}^{\infty}q_{\phi(x,s)}(f(\phi(x,s)))ds$ or $\int_{0}^{\infty}c(\phi(x,s),f(\phi(x,s))))ds$ is finite, then in the above inequality, the equality takes place; and if both $\int_{0}^{\infty}q_{\phi(x,s)}(f(\phi(x,s)))ds$ and $\int_{0}^{\infty}c(\phi(x,s),f(\phi(x,s))))ds$ are infinite, then the right hand side of the inequality is zero according to (1).

In the proof of part (a), it was observed that

[TABLE]

and

[TABLE]

are absolutely continuous in $t$ and are thus finite for each $t\in[0,\infty)$. As in the proof of part (a), similar calculations to those in (16) imply that for each $t\in[0,\infty),$

[TABLE]

where the last equality is by what was established in part (a). Therefore, for each $t\in[0,\infty),$

[TABLE]

where the inequality holds because $V^{\ast}(x)\geq 1$ for each $x\in S.$ Taking $\mathop{\underline{\lim}}_{t\rightarrow\infty}$ on the both sides of the previous equality yields:

[TABLE]

with the inequality following from (17). Hence

[TABLE]

Here it is clear that $s\in[0,\infty)\rightarrow f(\phi(x,s))$ can be identified as an element of ${\cal R}$, denoted as $\tilde{f}^{x}$. In fact, $\tilde{f}_{s}^{x}=\delta_{\{f(\phi(x,s))\}}$ for each $s\in[0,\infty)$, whereas $x\in S\rightarrow\tilde{f}^{x}\in{\cal R}$ is measurable. This measurable mapping $x\in S\rightarrow\tilde{f}^{x}\in{\cal R}$ defines a deterministic stationary optimal strategy for the risk-sensitive DTMDP problem (22) by Proposition A.1. It is clear that the measurable mapping $x\in S\rightarrow f(x)\in A$ defines an optimal deterministic stationary policy for the PDMDP problem (8).

Finally, we show the remaining part of (a). Let $H^{\ast}$ be a measurable $[1,\infty)$-valued function on $S$ such that

[TABLE]

There exists a measurable mapping $h$ from $S$ to $A$ such that

[TABLE]

c.f., Proposition D.5 of [17]. It follows that $\int_{0}^{s}\int_{S}H^{\ast}(y)\tilde{q}(dy|\phi(x,\tau),h(\phi(x,\tau)))d\tau$ is absolutely continuous in $s\in[0,t]$ for each $t\geq 0.$ As in the proof of part (b),

[TABLE]

and by passing to the lower limit as $t\rightarrow\infty$,

[TABLE]

It remains to refer to Proposition A.1 for that $H^{\ast}(x)\geq V^{\ast}(x)$ for each $x\in S.$ $\hfill\Box$

Proof of Theorem 3.2. Let $V^{\ast}_{0}(x):=1$ for each $x\in S.$ For each $n\geq 0,$ one can legitimately define

[TABLE]

Recall that the DTMDP model $\{\textbf{X},\textbf{A},p,l\}$ satisfies Condition A.1, as noted in the proof of Lemma 4.1. Then by Proposition A.1, $\{V_{n}^{\ast}\}$ is a monotone nondecreasing sequence of $[1,\infty)$-valued measurable functions on $S$ such that $V^{\ast}_{n}(x)\uparrow V^{\ast}(x)$ as $n\uparrow\infty,$ for each $x\in S.$

Let $n\geq 0$ be fixed. As in Lemma 4.3, for each $x\in S$, there is some $\rho^{\ast}\in{\cal R}$ such that

[TABLE]

Also the relevant version of Lemma 4.2 holds: for each $x\in S$ and $\rho\in{\cal R}$,

[TABLE]

is monotone nondecreasing in $t\in[0,\infty)$. Clearly, $V^{\ast}_{n+1}(\phi(x,t))$ is absolutely continuous in $t\in[0,\infty)$ for each $x\in S$.

Corresponding to (16), we now have

[TABLE]

where $\tau\in[0,t]\rightarrow U^{\ast}_{n+1}(x,\tau)$ is integrable and coincides with $\frac{\partial V^{\ast}_{n+1}(\phi(x,t))}{\partial t}$ almost everywhere, and $f$ is some measurable mapping from $S$ to $A$, whose existence is guaranteed by Proposition D.5 of [17]. Continued from the above relation, the reasoning in the proof of the first assertion in part (a) of Theorem 3.1 can be followed: eventually we see

[TABLE]

almost everywhere in $\tau\in[0,t],$ i.e., the equation

[TABLE]

is satisfied by $V=V^{\ast}_{n+1}.$

Recall that $V^{\ast}_{0}=V^{(0)}$. Suppose the recursive definition in (9) is valid up to step $n$, and $V^{\ast}_{n}(x)=V^{(n)}(x)$ for each $x\in S.$ Consider an arbitrarily fixed $[1,\infty)$-valued measurable solution $V$ to (20), and let $f^{\ast}$ be a measurable mapping from $S$ to $A$ such that

[TABLE]

One can follow the reasoning in the last part of the proof of Theorem 3.1, and see, c.f. (18),

[TABLE]

where the last equality is by (19). Thus, $V^{\ast}_{n+1}$ is the minimal $[1,\infty)$-valued measurable solution to (20), and coincides with $V^{(n+1)}$. Therefore, by induction $V^{\ast}_{n}=V^{(n)}$ for each $n\geq 0.$ It follows now that $V^{(n)}(x)\uparrow V^{\ast}(x)$ as $n\uparrow\infty$ for each $x\in S.$ $\hfill\Box$

5 Conclusion

In this paper, we considered total undiscounted risk-sensitive PDMDP in Borel state and action spaces with a nonnegative cost rate. The transition and cost rates are assumed to be locally integrable along the drift. Under quite natural conditions, we showed that the value function is a solution to the optimality equation, justified the value iteration algorithm, and showed the existence of deterministic stationary optimal policy. As a corollary, the obtained results were applied to improving significantly known results for finite horizon undiscounted and infinite horizon discounted risk-sensitive CTMDP in the literature.

Appendix A Appendix

For ease of reference, we present the relevant notations and facts about the risk-sensitive problem for a DTMDP. The proofs of the presented statements can be found in [16] or [27]. Standard description of a DTMDP can be found in e.g., [17, 22].

Consider a discrete-time Markov decision process with the following primitives:

• •

X is a nonempty Borel state space.

• •

A is a nonempty Borel action space.

• •

$p(dy|x,a)$ is a stochastic kernel on ${\cal B}(\textbf{X})$ given $(x,a)\in\textbf{X}\times\textbf{A}$.

• •

$l$ a $[0,\infty]$-valued measurable cost function on $\textbf{X}\times\textbf{A}\times\textbf{X}.$

Let $\Sigma$ be the space of strategies, and $\Sigma_{DM}$ be the space of all deterministic strategies for the DTMDP. Let the controlled and controlling processes be denoted by $\{Y_{n},n=0,1,\dots,\infty\}$ and $\{A_{n},n=0,1,\dots,\infty\}$, respectively. The strategic measure of a strategy $\sigma$ given the initial state $x\in\textbf{X}$ is denoted by $\textbf{P}_{x}^{\sigma}$. The expectation taken with respect to $\textbf{P}_{x}^{\sigma}$ is denoted by $\textbf{E}_{x}^{\sigma}.$

Consider the optimal control problem

[TABLE]

It is also referred to as the risk-sensitive DTMDP problem. We denote the value function of problem (21) by $\textbf{V}^{\ast}$. Then a strategy $\sigma^{\ast}$ is called optimal for problem (21) if $\textbf{V}(x,\sigma^{\ast})=\textbf{V}^{\ast}(x)$ for each $x\in\textbf{X}.$

Condition A.1

• (a)

The function $l(x,a,y)$ is lower semicontinuous in $a\in\textbf{A}$ for each $x,y\in\textbf{X}.$

• (b)

For each bounded measurable function $f$ on X and each $x\in\textbf{X},$ $\int_{\textbf{X}}f(y)p(dy|x,a)$ is continuous in $a\in\textbf{A}.$

• (c)

The space A is a compact Borel space.

Proposition A.1

Suppose Condition A.1 is satisfied.

• (a)

The value function $\textbf{V}^{\ast}$ is the minimal $[1,\infty]$-valued measurable solution to

[TABLE]

• (b)

Let U be a $[1,\infty]$-valued lower semianalytic function on X. If

[TABLE]

then $\textbf{U}(x)\geq\textbf{V}^{\ast}(x)$ for each $x\in\textbf{X}.$ In particular, if the function U satisfying the above relation is $[1,\infty)$-valued, then so is the value function $\textbf{V}^{\ast}.$

• (c)

Let $\varphi$ be a deterministic stationary strategy for the DTMDP model $\{\textbf{X},\textbf{A},p,l\}$. If

[TABLE]

then $\textbf{V}^{\ast}(x)=\textbf{V}(x,\varphi)$ for each $x\in\textbf{X}.$

• (d)

Let $\textbf{V}^{(0)}(x):=1$ for each $x\in\textbf{X}$, and for each $n=1,2,\dots,$

[TABLE]

Then $(\textbf{V}^{(n)}(x))$ increases to $\textbf{V}^{\ast}(x)$ for each $x\in\textbf{X}$, where $\textbf{V}^{\ast}$ is the value function for problem (21). Furthermore, there exists a deterministic stationary strategy $\varphi$ satisfying (23), and so in particular, there exists a deterministic stationary optimal strategy for the risk-sensitive DTMDP problem (21).

Acknowledgement. We thank the referees for their remarks, which improved the presentation of this paper. This work is partially supported by a grant from the Royal Society (IE160503).