Long-run risk sensitive dyadic impulse control

TL;DR

This paper advances long-run risk-sensitive impulse control for unbounded, non-uniformly ergodic Markov processes by integrating geometric drift and local minorisation, establishing solution existence for Bellman equations.

Contribution

It extends the local span-contraction approach to unbounded processes using weight norms, addressing a gap in the literature on risk-sensitive control.

Findings

Established existence of solutions to Bellman equations for unbounded processes

Extended geometric drift and minorisation techniques to risk-sensitive impulse control

Provided examples of processes satisfying the new assumptions

Abstract

In this paper long-run risk sensitive optimisation problem is studied with dyadic impulse control applied to continuous-time Feller-Markov process. In contrast to the existing literature, focus is put on unbounded and non-uniformly ergodic case by adapting the weight norm approach. In particular, it is shown how to combine geometric drift with local minorisation property in order to extend local span-contraction approach when the process as well as the linked reward/cost functions are unbounded. For any predefined risk-aversion parameter, the existence of solution to suitable Bellman equation is shown and linked to the underlying stochastic control problem. For completeness, examples of uncontrolled processes that satisfy the geometric drift assumption are provided.

Equations

$$J_T(x,V)$$

$$J(x,V):=\underset{n\to \infty }{\mathrm{liminf}}\frac{J_{T_n}(x,V)}{T_n}.$$

$${\mu }_{(x,V)}^{\gamma }(Z):=\{\begin{array}{ll}1/\gamma \ln {\mathbb{E}}_{(x,V)}\left[\exp (\gamma Z)\right]& \gamma \ne 0,\\ {\mathbb{E}}_{(x,V)}\left[Z\right]& \gamma =0,\end{array}$$

$$J_T(x,V)$$

$$\Vert g{\Vert }_{\omega }:=\underset{x\in E}{\sup }\frac{|g(x)|}{1+\omega (x)}<\infty .$$

$${\mu }_x^{\gamma }\left({\int }_0^{\delta }\omega (X_s)ds\right)\le \omega (x)+M_1(\gamma )\text{ and }{\mu }_x^{\gamma }\left(\omega (X_{\delta })\right)\le b_1\omega (x)+M_2(\gamma ).$$

$$\underset{x\in C_R}{\inf }{\mathbb{P}}_x[X_{\delta }\in A]\ge d\nu (A),A\in \mathcal{B}(E),$$

$$d{X}_t=[a{X}_t+g(X_t)]dt+\sigma (X_t)d{W}_t,$$

$$\underset{V\in {\mathbb{V}}_{\delta }}{\sup }J(x_0,V),$$

$$w_{\delta }^{\gamma }(x)+{\lambda }_{\delta }^{\gamma }=\max \left\{{\mu }_x^{\gamma }\left({\int }_0^{\delta }f(X_s)ds+w_{\delta }^{\gamma }(X_{\delta })\right),\underset{\xi \in U}{\sup }\left({\mu }_{\xi }^{\gamma }\left({\int }_0^{\delta }f(X_s)ds+w_{\delta }^{\gamma }(X_{\delta })\right)+c(x,\xi )\right)\right\},$$

$$w_{\delta }^{\gamma }(x)+{\lambda }_{\delta }^{\gamma }=\underset{a\in \bar{U}}{\sup }\left({\mu }_x^{\gamma ,a}\left({\int }_0^{\delta }f(X_s)ds+w_{\delta }^{\gamma }(X_{\delta })\right)+\bar{c}(x,a)\right),$$

$$a=(a^1,a^2),a^1\in \{0,1\},a^2\in U,\bar{U}=\{0,1\}\times U,$$

$$\bar{c}(x,a)=\left\{\begin{array}[]{lll}0&\mbox{if}&a^{1}=0,\\ c(x,\xi)&\mbox{if}&a^{1}=1\,,\,a^{2}=\xi,\\ \end{array}\right.$$

$$\mu_{x}^{\gamma,a}=\left\{\begin{array}[]{lll}\mu^{\gamma}_{\xi}&\quad\mbox{if}&\quad a=(1,\xi),\\ \mu^{\gamma}_{x}&\quad\mbox{if}&\quad a=(0,\xi).\\ \end{array}\right.$$

$$R_{\gamma }g(x):=\underset{a\in \bar{U}}{\sup }\left({\mu }_x^{\gamma ,a}\left({\int }_0^{\delta }f(X_s)ds+g(X_{\delta })\right)+\bar{c}(x,a)\right),g\in {\mathcal{C}}_{\omega }(E),$$

$$T_{\gamma }g(x):=\gamma R_{\gamma }(g(x)/\gamma ).$$

$${\mu }_{(x,g,a)}^{\ast }(B):=\frac{{\mathbb{E}}_x^a\left[e^{\gamma {\int }_0^{\delta }f(X_s)ds+g(X_{\delta })}\text{mathbbm}{1}_{\{X_{\delta }\in B\}}\right]}{{\mathbb{E}}_x^a\left[e^{\gamma {\int }_0^{\delta }f(X_s)ds+g(X_{\delta })}\right]},$$

$$\mathbb{E}_{x}^{a}:=\mu_{x}^{0,a}=\left\{\begin{array}[]{lll}\mathbb{E}_{\xi}&\quad\mbox{if}&\quad a=(1,\xi),\\ \mathbb{E}_{x}&\quad\mbox{if}&\quad a=(0,\xi).\\ \end{array}\right.$$

$$F(x)$$

$$\le {\mu }_x^{\gamma }\left(\Vert f{\Vert }_{\omega }{\int }_0^{\delta }\omega (X_s)ds+\Vert g{\Vert }_{\omega }\omega (X_{\delta })\right)+(\delta \Vert f{\Vert }_{\omega }+\Vert g{\Vert }_{\omega }).$$

$${\mu }_x^{\gamma }\left(\Vert f{\Vert }_{\omega }{\int }_0^{\delta }\omega (X_s)ds+\Vert g{\Vert }_{\omega }\omega (X_{\delta })\right)$$

$$\le (\Vert f{\Vert }_{\omega }+\Vert g{\Vert }_{\omega })\left[\omega (x)+M_1(\gamma \Vert f{\Vert }_{\omega }/2)+M_2(-\gamma \Vert g{\Vert }_{\omega })\right].$$

$$\Vert F{\Vert }_{\omega }<\infty .$$

$$\underset{\xi \in U}{\sup }\left(F(\xi )+c(x,\xi )\right)\le \Vert F{\Vert }_{\omega }\left(\underset{\xi \in U}{\sup }\omega (\xi )+1\right)+c_0<\infty .$$

$$\underset{x\in E}{\inf }\frac{R_{\gamma }g(x)}{1+\omega (x)}\ge \underset{x\in E}{\inf }\frac{F(x)}{1+\omega (x)}>-\infty ,$$

$$Z(n,m):=e^{{\gamma }_n\left[{\int }_0^{\delta }{f}_m(X_s)ds+g_m(X_{\delta })\right]},$$

$$\left|({\gamma }_n-\gamma )\left[{\int }_0^{\delta }{f}_m(X_s)ds+g_m(X_{\delta })\right]\right|\le (\delta +1)m\left|{\gamma }_n-\gamma \right|$$

$${\mathbb{E}}_{x_n}\left[e^{{\gamma }_n\left[{\int }_0^{\delta }{f}_m(X_s)ds+g_m(X_{\delta })\right]}\right]\to {\mathbb{E}}_x\left[e^{\gamma \left[{\int }_0^{\delta }{f}_m(X_s)ds+g_m(X_{\delta })\right]}\right],n\to \infty ,$$

$${\mathbb{E}}_{x_n}\left[Z(n,m)\right]\to {\mathbb{E}}_x\left[Z(\infty ,m)\right],n\to \infty .$$

$${\mathbb{E}}_y\left[(Z(n,m){)}^2\right]$$

$$\le {\mathbb{E}}_y\left[e^{-2\bar{\gamma }\left[\Vert f{\Vert }_{\omega }{\int }_0^{\delta }(\omega (X_s)+1)ds+\Vert g{\Vert }_{\omega }(\omega (X_{\delta })+1)\right]}\right]$$

$$=e^{-2\bar{\gamma }{\mu }_y^{-2\bar{\gamma }}\left(\Vert f{\Vert }_{\omega }{\int }_0^{\delta }(\omega (X_s)+1)ds+\Vert g{\Vert }_{\omega }(\omega (X_{\delta })+1)\right)},$$

Full text

Long-run risk sensitive dyadic impulse control

Marcin Pitera Institute of Mathematics, Jagiellonian University, Cracow, Poland,

      Email: marcin.pitera@im.uj.edu.pl, research supported by NCN grant 2016/23/B/ST1/00479.

  

and   Łukasz Stettner

Institute of Mathematics, Polish Academy of Sciences, Warsaw, Poland,

      Email: l.stettner@impan.pl, research supported by NCN grant 2016/23/B/ST1/00479.

( This version: )

[TABLE]

1 Introduction

Let $(\Omega,\mathcal{F},\mathbb{F},\mathbb{P})$ be a continuous-time filtered probability space that satisfy the usual conditions. In particular, we assume that $\mathbb{F}=\{\mathcal{F}_{t}\}_{t\in\mathbb{T}}$, where $\mathbb{T}=\mathbb{R}_{+}$, $\mathcal{F}_{0}$ is trivial, and $\mathcal{F}=\bigcup_{t\in\mathbb{T}}\mathcal{F}_{t}$. Moreover, let $X=(X_{t})$ be a Feller-Markov process with values in a locally compact space $E$; for simplicity we set $E=\mathbb{R}^{d}$ but most results transfer directly to the general case. The process $X$ is controlled by impulses of the form $(\tau,\xi)$: at random time $\tau$ the process is shifted from the state $X_{\tau}$ to the state $\xi$ and follows its dynamics until the next impulse. We assume that the shift $\xi$ takes values in a compact set $U\subseteq E$. Let $\mathbb{V}$ be a space of all admissible impulse control strategies $V=\{(\tau_{i},\xi_{i})\}_{i=1}^{\infty}$, i.e. sequences of strictly increasing (possibly infinite) Markov times $\tau_{i}$ and shift random variables $\xi_{i}$. Assuming that $X_{0}=x$ (where $x\in E$) and $V\in\mathbb{V}$ we use $(\hat{\Omega},\hat{\mathcal{F}},\mathbb{P}_{(x,V)})$ to denote the probability space related to the corresponding controlled process $X$. For brevity, we omit the construction of this space; see Robin (1978) for details. We refer to Palczewski & Stettner (2017) where a similar impulse control framework is considered and discussed in details; see also Stettner (1982, 1989).

The main goal of this paper is to study risk sensitive impulse control problem with reward and shift cost functions embedded in the objective function. We consider long-run version of the risk sensitive criterion with risk aversion parameter $\gamma<0$ given by

[TABLE]

defined for all $T\in\mathbb{T}$, $x\in E$ and admissible controls $V\in\mathbb{V}$; note that the process $X$ has initial state $x$ and it’s dynamics depends on control $V$. In (1.1), the function $c\colon E\times U\to\mathbb{R}_{-}$ relates to the shift execution cost function, the function $f\colon E\to\mathbb{R}$ corresponds to the reward function, and $X_{\tau_{i}^{-}}$ is the state of the process before the $i$-th impulse (with a natural meaning if there is more than one impulse at the same time).

Risk sensitive control could be seen as a non-linear extension of the risk-neutral expected cost per unit of time control studied e.g. in Robin (1981, 1983); see Palczewski & Stettner (2017) for a more recent contribution in the impulse control context. While impulse control is among the most popular forms of control, application of the standard methods in the risk sensitive case usually lead to difficult problems linked to quasi variational inequalities; see Nagai (2007), and references therein. Consequently, alternative tools need to be developed; see e.g. Hdhiri & Karouf (2011). In this paper, we refine and extend the probabilistic approach to impulse risk sensitive control developed initially in Sadowy & Stettner (2002) by allowing unbounded value/cost functions and non-uniform ergodicity of the underlying process. For more general background on long-run risk-sensitive control in the bounded framework see e.g. Fleming & McEneaney (1995) or Di Masi & Stettner (1999).

We focus on the dyadic impulse control strategies where the shifts can be applied on a discrete $\delta$-dyadic time grid. By considering weighted norms, we expand the framework initiated in Hairer & Mattingly (2011) and Pitera & Stettner (2016); we also refer to Shen et al. (2013); Bäuerle & Rieder (2017), and references therein. Our approach is based on the span-contraction framework, with generic set of assumptions centred around geometric drift and local minorisation; for (alternative) vanishing discount approach see e.g. Cavazos-Cadena & Hernández-Hernández (2017) and references therein. Apart from extending the span-contraction approach to the unbounded case, we also show the simple novel long-run noise control method based on application of Hölder’s inequality to the underlying entropic utility. By splitting the process into different components, and applying the entropic super and subadditive bounds (see Lemma 6.1) we are able to get rid of the noise in the limit. This simple observation allow us to quickly link the Bellman solution to the underlying optimisation problem when the noise is unbounded; see Proposition 4.3. This method is quite general and could be used e.g. in long-run risk-sensitive portfolio optimisation. As an example, on can easily refine Proposition 5 in Pitera & Stettner (2016) by showing that Bellman equation always corresponds to the optimal strategy (defined therein) without any additional assumptions.

This paper is organised as follows. Section 2 establishes the general setup. In particular, we introduce and discuss core assumptions and state the main problem therein. In Section 3, we introduce the dyadic Bellman equation and show that the solution to it exists. Theorem 3.2 stating that the Bellman operator is a local contraction in the shrinked $\omega$-span norm is a central part of the span-contraction approach and might be seen as one of the main results of this paper. Section 4 links the Bellman’s equation to the corresponding dyadic optimal control problem (2.5); the main result of this section is Proposition 4.3. In Section 5, we show the reference examples of uncontrolled processes that satisfies entropic inequalities that will be introduced in Assumption ((A.3)); this is important from the pragmatic point of view perspective, as the assumption might look restrictive at the first sight. Finally, in Appendix 6 we introduce and prove some supplementary results including the simple proof of entropic Hölder’s inequalities.

2 Preliminaries

Let us fix $\delta>0$ ane let $\mathbb{T}_{\delta}:=\{n\delta\}_{n\in\mathbb{N}}$ denote the related $\delta$-dyadic time grid. We use $\mathbb{V}_{\delta}\subset\mathbb{V}$ to denote the space of all related dyadic impulse control strategies; see Sadowy & Stettner (2002) for details. For transparency, for a fixed $\gamma<0$ any $n\in\mathbb{N}$, we define $T_{n}:=n\delta$ and consider the dyadic average-cost long-run version of (1.1) defined as

[TABLE]

While most results could be easily extended to the full time domain, considering only discrete dyadic times in (2.1) increases the transparency and is more natural when considering discrete Bellman equations; when required, we provide additional comments on how to extend our framework to full time domain.

Given the initial state $x\in E$ and impulsive control $V\in\mathbb{V}$, we define the corresponding entropic utility measure $\mu^{\gamma}_{(x,V)}:L^{0}(\hat{\Omega},\hat{\mathcal{F}},\mathbb{P}_{(x,V)})\to\bar{\mathbb{R}}$ with risk-aversion parameter $\gamma\in\mathbb{R}$ by setting

[TABLE]

where $\mathbb{E}_{(x,V)}$ is the expectation operator corresponding to $\mathbb{P}_{(x,V)}$. For brevity, we use $\mu_{x}^{\gamma}$ to denote entropic utility corresponding to uncontrolled process starting at $x\in E$ (e.g. for $V\in\mathbb{V}$ such that $\tau_{i}=\infty$ for $i\in\mathbb{N}$). If there is no ambiguity, we write $\mu^{\gamma}$ instead of $\mu^{\gamma}_{x}$ or $\mu^{\gamma}_{(x,V)}$. Same applies to the probability measure $\mathbb{P}_{(x,V)}$ as well as the expectation operator $\mathbb{E}_{(x,V)}$. In particular, note that (1.1) could be rewritten as

[TABLE]

Let $\omega:E\to\mathbb{R}_{+}$ be a fixed continuous weight function and let $C_{\omega}(E)$ denote the space of all real-valued continuous functions which are bounded wrt. $\omega$-norm, i.e. functions $g\colon E\to\mathbb{R}$ such that

[TABLE]

Next, we present assumptions that will be used throughout the paper. In assumptions ((A.3))–((A.4)) the process $X=(X_{t})$ corresponds to the uncontrolled process with initial state $x\in E$.

1. (A.1)

(Reward function constraints.) The function $f$ is continuous and $\|f\|_{\omega}<\infty$. 2. (A.2)

(Shift cost function constraints.) The function $c$ is continuous and there exists $c_{0}<0$, such that for all $x\in E$ and $\xi\in U$ we get $c(x,\xi)\leq c_{0}$. Moreover, $\|\hat{c}\|_{\omega}<\infty$, where $\hat{c}\colon E\to\mathbb{R}_{-}$ is given by $\hat{c}(x):=\inf_{\xi\in U}c(x,\xi)$. 3. (A.3)

(Geometric drift with controllable noise.) There exist a constant $b_{1}\in(0,1)$, and (finite) functions $M_{1},M_{2}\colon\mathbb{R}\to\mathbb{R}$, such that for any $\gamma\in\mathbb{R}$ and $x\in E$ we get

[TABLE] 4. (A.4)

(Local minorization.) For any $R>0$, there exists $d>0$ and probability measure $\nu$, such that

[TABLE]

where $C_{R}=\{x\in E\colon\omega(x)\leq R\}$ and $\nu$ satisfies $\nu(U)>0$.

Let us now briefly discuss the assumptions.

Assumptions ((A.1)) and ((A.2)) are standard assumptions which allow us to operate on the space $C_{\omega}(E)$ of $\omega$-bounded functions. For technical reasons, we assume that the cost of the shift is always strictly negative and bounded away from zero (by $c_{0}$); this is a classical impulse control assumption.

Assumption ((A.3)) relates to geometric drift property of the uncontrolled process. For simplicity, let us focus on the second inequality. For a fixed $x\in E$ the random variable $\omega(X_{\delta})-b_{1}\omega(x)$ might be understood as the $\omega$-noise, with upper bound imposed on its entropic utility. Since the distribution of $\omega(X_{\delta})-b_{1}\omega(x)$ might depend on $x\in E$ we cannot split noise from the starting point as done in Pitera & Stettner (2016); the global upper bound $M_{2}(\gamma)$ in (2.3) relates to distribution level constraints. Indeed, assuming the standard probability space and noting that entropic risk measure is law-invariant, we can rephrase ((A.3)) using the concept of first-order stochastic dominance: we can assume existence of a random variable $Z$, such that $Z$ has finite moments and stochastically dominates (positive part of) $\omega(X_{\delta})-b_{1}\omega(x)$ for any $x\in E$; see (Bäuerle & Müller, 2006, Theorem 4.2) for details. In order to have all moments finite $Z$ must belong to Orlicz heart induced by the entropic risk measure; see Cheridito & Li (2009). For example, with $E=\mathbb{R}$ and $\omega(\cdot)=|\cdot|$ assumption ((A.3)) holds for uncontrolled processes with dynamics given by

[TABLE]

where $a<0$, functions $g\colon\mathbb{R}\to\mathbb{R}$ and $\sigma\colon\mathbb{R}\to\mathbb{R}_{+}$ are bounded, and $W_{t}$ is a standard Brownian motion. More generally, ((A.3)) is satisfied for Gaussian-type of noise given e.g. via suprema of Gaussian random vectors; we refer to Section 5 for more details and to Pitera & Stettner (2016) for further discussion.

Assumption ((A.4)) is a (local) minorization property. Combined with ((A.3)) it constitues the ergodicity property of the underlying uncontrolled process; see Hairer & Mattingly (2011) for details. For bounded $\omega$ it is equivalent to a global Doeblin’s condition (uniform ergodicity), while for unbounded $\omega$ it might be linked to the local mixing condition. Note that we additionally require that the support of invariant measure $\nu$ must have a non-empty intersection with control (shift) set $U$.

The main goal of this paper is to find optimal control (and solution) to problem

[TABLE]

where $x_{0}$ is the (given) initial state.

Remark 2.1* (Dyadic dynamics).*

While in this paper we fix time-step $\delta>0$, it might be interesting to extend the assumptions for the general dyadic control case. First, note that assumptions ((A.1)) and ((A.2)) are independent of the underlying choice of $\delta$. Second, assumption ((A.3)) relies on the choice of $\delta$ e.g. via the shrinkage constant $b_{1}$ and noise constraints $M_{i}(\gamma)$ ($i=1,2$). Treating $b_{1}$ and $M_{i}(\gamma)$ as functions of $\delta$ and letting $\delta\to 0$ we should get $b_{1}(\delta)\to 1$ and $M_{i}(\gamma,\delta)\to 0$, for any $\gamma\in\mathbb{R}$. Also, assuming the noise is divisible, it would be rational to assume $\limsup_{\delta\to\infty}M_{i}(\gamma,\delta)/\delta<\infty$. Finally, note that assumption ((A.4)) depends on the choice of the time-grid parameter, but it would be (typically) enough to introduce dependence of $d$ and $\nu$ on $\delta$, without any additional uniform constraints.

3 Bellman equation

Following Sadowy & Stettner (2002) and Pitera & Stettner (2016) we define the Bellman equation for the dyadic impulsive control as

[TABLE]

for $x\in E$, where $\lambda^{\gamma}_{\delta}\in\mathbb{R}$ and $w^{\gamma}_{\delta}\in C_{\omega}(E)$. Equation (3.1) can be equivalently stated as the ordinary risk sensitive discrete-time control problem

[TABLE]

where

[TABLE]

[TABLE]

[TABLE]

On the space $C_{\omega}(E)$ , we define the corresponding discrete-time Bellman operator

[TABLE]

and the associated operator

[TABLE]

For any $g\in C_{\omega}(E)$ and $x\in E$ we use $a_{(x,g)}$ to denote the maximiser of $T_{\gamma}g(x)$. Recalling that the Esscher transformation defines the maximising measure in the robust (dual, biconjugate) representation of the entropic utility measure (see e.g. Dai Pra et al. (1996)) for any $g\in C_{\omega}(E)$ , $x\in E$ , $a\in\bar{U}$ , and measurable set $B$, we define the associated measure

[TABLE]

where

[TABLE]

For more details we refer to Pitera & Stettner (2016) where the equivalent of (3.4) is defined in Equation (29) and the dual representation of entropic utility in similar setting is discussed; see also Gerber (1979) for more details about Esscher transform.

Proposition 3.1**.**

Under assumptions ((A.1))–((A.3)) operators $R_{\gamma}$ and $T_{\gamma}$ transforms the set $\mathcal{C}_{\omega}(E)$ into itself. Moreover, for any $g\in\mathcal{C}_{\omega}(E)$ the mappings $(x,\gamma)\mapsto T_{\gamma}g(x)$ and $(x,\gamma)\mapsto R_{\gamma}g(x)$ are continuous on $E\times(-\infty,0)$.

Proof.

We only show the proof for $R_{\gamma}$ as the proof for $T_{\gamma}$ is analogous. Let $\gamma<0$ and $g\in\mathcal{C}_{\omega}(E)$.

First, let us prove that $\|R_{\gamma}g\|_{\omega}<\infty$. For $x\in E$ we set $F(x):=\mu^{\gamma}_{x}\left(\int_{0}^{\delta}f(X_{s})\,ds+g(X_{\delta})\right)$. Using ((A.1)), ((A.3)), and monotonicity of the entropic utility measure, for any $x\in E$ we get

[TABLE]

Now, using ((A.3)) and Hölder’s inequality for entropic utility measure with $p=2$ (see Lemma 6.1), we know that for any $x\in E$ we get

[TABLE]

and consequently $\sup_{x\in E}\frac{F(x)}{1+\omega(x)}<\infty$. Similarly, one can show that $\inf_{x\in E}\frac{F(x)}{1+\omega(x)}>-\infty$. Thus, we get

[TABLE]

Now, noting that $\omega$ is continuous and $U$ is compact, for any $x\in E$ we get

[TABLE]

Combining (3.7) with (3.8) we get $\sup_{x\in E}\frac{R_{\gamma}g(x)}{1+\omega(x)}<\infty$. Then, noting that $R_{\gamma}g(x)\geq F(x)$, we get

[TABLE]

which concludes the proof of $\|R_{\gamma}g\|_{\omega}<\infty$.

Second, let us prove that the mapping $(x,\gamma)\mapsto R_{\gamma}g(x)$ is continuous on $E\times(-\infty,0)$. Fix $\gamma<0$, $x\in E$, and let $\{(x_{n},\gamma_{n})\}_{n\in\mathbb{N}}$ be a sequence satisfying $(x_{n},\gamma_{n})\to(x,\gamma)$, $n\to\infty$, where for any $n\in\mathbb{N}$ we have $(x_{n},\gamma_{n})\in E\times(-\infty,0)$. For $n,m\in\mathbb{N}\cup\{\infty\}$ we set

[TABLE]

where $f_{m}\colon E\to\mathbb{R}$ and $g_{m}\colon E\to\mathbb{R}$ are given by $f_{m}(\cdot)=(f(\cdot)\vee-m)\wedge m$ and $g_{m}(\cdot)=(g(\cdot)\vee-m)\wedge m$, and notation $\gamma_{\infty}:=\gamma$, $f_{\infty}(\cdot):=f(\cdot)$, and $g_{\infty}(\cdot):=g(\cdot)$ is used. Clearly, $f_{m}(z)\to f(z)$ and $g_{m}(z)\to g(z)$ for $z\in E$, as $m\to\infty$. For any $m\in\mathbb{N}$, combining Feller property with the fact that

[TABLE]

and $|\gamma_{n}-\gamma|\to 0$, as $n\to\infty$, we get

[TABLE]

which could be rewritten as

[TABLE]

Next, we show that the class of random variables $\{Z(n,m)\}_{n,m\in\mathbb{N}\cup\{\infty\}}$ is uniformly integrable on $\mathbb{P}_{y}$, for any $y\in\hat{V}$, where $\hat{V}\subset E$ is a compact set such that $(\{x_{n}\}_{n\in\mathbb{N}}\cup\{x\}\cup U)\subseteq\hat{V}$. Using ((A.3)), for any $y\in\hat{V}$ and $m,n\in\mathbb{N}\cup\{\infty\}$, we get

[TABLE]

where $\bar{\gamma}:=\inf_{n\in\mathbb{N}}\gamma_{n}$. By similar arguments as in the first part of the proof (i.e. using ((A.3)) and Hölder’s inequalities for entropic utility measure), recalling that $\omega$ is continuous, and $\hat{V}$ is compact we get

[TABLE]

Combining (3.10) with (3.11), and noting the upper bound in (3.10) is independent of $n$ and $m$, we get that the class $\{Z(n,m)\}_{n,m\in\mathbb{N}\cup\{\infty\}}$ is $L^{2}$-bounded sequence on $\mathbb{P}_{y}$, for any $y\in\hat{V}$. In particular, this implies uniform integrability of $\{Z(n,m)\}_{n,m\in\mathbb{N}\cup\{\infty\}}$ on $\mathbb{P}_{y}$ and

[TABLE]

for $y\in\hat{V}$ and $n\in\mathbb{N}\cup\{\infty\}$. Moreover, since the upper $L^{2}$-bound in (3.10) could be chosen independently of $y$ we get

[TABLE]

Next, to show that

[TABLE]

it is enough to note that for any fixed $m\in\mathbb{N}$ we get

[TABLE]

Indeed, combining (3.9), (3.12), (3.13), with (3.15), and letting $m\to\infty$, we get (3.14), i.e. property

[TABLE]

which in turn implies $\tilde{Z}(x_{n},\gamma_{n})\to\tilde{Z}(x,\gamma)$, where $\tilde{Z}(w,z):=\mu_{w}^{z}\left(\int_{0}^{\delta}f(X_{s})\,ds+g(X_{\delta})\right)$. Next, noting that for any $\xi\in U$ we get $\tilde{Z}(\xi,\gamma_{n})\to\tilde{Z}(\xi,\gamma)$, and $U$ is compact, we get

[TABLE]

from which continuity of $(x,\gamma)\mapsto R_{\gamma}g(x)$ follows. ∎

We now show that on $C_{\omega}(E)$ the operator $T_{\gamma}$ is a local contraction under the suitable span-norm. To ensure that property for each single step we need to shrink the original $\omega$-norm. For any $\beta>0$ the shrinked norm $\|\cdot\|_{\beta,\omega}$ is given by

[TABLE]

while the corresponding span semi-norm is defined as

[TABLE]

It is useful to note that for any $g\in C_{\omega}(E)$ and $\beta>0$ we get

[TABLE]

so that the span $\omega$-norm could be considered as the centered (wrt. 0) $\omega$-norm; see (Pitera & Stettner, 2016, Section 3) and (Hairer & Mattingly, 2011, Section 2) for details.

Theorem 3.2**.**

Let $\gamma<0$. Under assumptions ((A.1))–((A.4)), for sufficiently small $\beta>0$, the operator $T_{\gamma}$ is a local contraction under $\|\cdot\|_{\beta,\omega\textrm{-span}}$, i.e. there exist functions $\beta:\mathbb{R}_{+}\to(0,1)$ and $L:\mathbb{R}_{+}\to(0,1)$ such that

[TABLE]

for $f_{1},f_{2}\in\mathcal{C}_{\omega}(E)$, such that $\|f_{1}\|_{\omega\textrm{-span}}\leq M$ and $\|f_{2}\|_{\omega\textrm{-span}}\leq M$.

Proof.

For brevity, we present only the outline the proof; please see (Pitera & Stettner, 2016, Theorem 1) for more details. The proof will be based on three steps.

Step 1)  We prove that for any $g_{1},g_{2}\in C_{\omega}(E)$ and $x,y\in E$ we get

[TABLE]

where

[TABLE]

$\bar{\mu}^{*}_{(\cdot)}$ is the projection of measure $\mu^{*}_{(\cdot)}$ (given in (3.4)) on the set of values of the processes $X$, $\|\cdot\|_{\beta,\omega\textrm{-var}}$ is the weighted total variation norm given by

[TABLE]

and $|\mathbb{H}|$ is the total variation of measure $\mathbb{H}$; see (Pitera & Stettner, 2016, Section 3) for details.

First, following the proof of (Pitera & Stettner, 2016, Lemma 1) (see also (Di Masi & Stettner, 1999, Proposition 2.2) where similar calculations are done for Bellman operator without impulse cost for any $g_{1},g_{2}\in C_{\omega}(E)$ and $x,y\in E$ we get

[TABLE]

Second, using (Pitera & Stettner, 2016, Proposition 2), we know there exists $d\in\mathbb{R}$ such that

[TABLE]

where

[TABLE]

Noting that

[TABLE]

and using the Hahn-Jordan decomposition for signed measure $\mathbb{H}^{g_{1},g_{2}}_{x,y}$, we get

[TABLE]

where $A$ corresponds to positive set of measure $\mathbb{H}^{g_{1},g_{2}}_{x,y}$. Consequently, recalling (3.17) and combining (3.19) with (3.20) we get (3.16).

Step 2)  We prove that for any fixed $M>0$ and $\phi\in(b_{1},1)$, there exists $\alpha_{\phi}>0$, such that

[TABLE]

for $x,y\in E$ and $g_{1},g_{2}\in\mathcal{C}_{\omega}(E)$ satisfying $\|f\|_{\omega\textrm{-span}}\leq M$ and $\|g\|_{\omega\textrm{-span}}\leq M$; $\|\cdot\|_{\textrm{var}}$ denotes the standard variation norm.

First, note that

[TABLE]

Consequently, it is enough to show that there exists $\alpha_{\phi}>0$ such that for any $x\in E$ , $a\in\bar{U}$ , and $g\in C_{\omega}(E)$ satisfying $\|g\|_{\omega\textrm{-span}}\leq M$, we get

[TABLE]

note that for $a\in\{1\}\times U$ the term $\phi\omega(x)$ is added artificially for consistency purposes and does not relate to state after applying the shift, i.e. since $\omega$ is bounded on the compact set $U$, for any $\xi\in U$ the term $\phi\omega(\xi)$ could be included in $\alpha_{\phi}$ by increasing the constant by $\phi\sup_{\xi\in U}\omega(\xi)$. Thus, setting $Z:=\gamma\int_{0}^{\delta}f(X_{s})\,ds+g(X_{\delta})$, recalling (3.4), and noting that it is sufficient to consider $a\in\{0\}\times U$ since $U\subset E$, we can rewrite inequality (3.22) as

[TABLE]

Let $K:=M-\delta\gamma\|f\|_{\omega}$. Multiplying both sides of (3.23) by $\frac{2K}{\phi-b_{1}}$, noting that $y<e^{y}$ for $y\in\mathbb{R}$, and taking logarithm on both sides it is enough to show

[TABLE]

which is equivalent to

[TABLE]

where $d\in\mathbb{R}$ is (centralizing constant) such that $\|g+d\|_{\omega}\leq M$. Noting that

[TABLE]

using Hölder’s inequality for entropic utility measure with $p=2$ (see Lemma 6.1), and recalling $\eqref{A.3}$ we get

[TABLE]

Similarly,

[TABLE]

where

[TABLE]

Combining (3.35), (3.26), and (3.27) with (3.24) we know it is enough to choose (large) $\alpha_{\phi}$ satisfying

[TABLE]

This concludes the proof of (3.21).

Step 3)  Finally, we want to show that for any fixed $M>0$, $\phi\in(b_{1},1)$ and $\alpha_{\phi}>0$, there exists $\beta\in(0,1)$ and $L\in(0,1)$ such that

[TABLE]

for any $x,y\in E$ and $g_{1},g_{2}\in\mathcal{C}_{\omega}(E)$ satisfying $\|f\|_{\omega\textrm{-span}}\leq M$ and $\|g\|_{\omega\textrm{-span}}\leq M$.

Let us fix $M>0$, $\phi\in(b_{1},1)$ and $\alpha_{\phi}>0$, and consider $R\in\mathbb{R}$ such that $R>\frac{2\alpha_{\phi}}{1-\phi}$. If $x,y\in E$ are such that $\omega(x)+\omega(y)>R$ then one could show that for any $\beta<1$ and

[TABLE]

the inequality (3.28) will hold; see proof of (Pitera & Stettner, 2016, Lemma 3) for details. On the other hand, if $x,y\in E$ are such that $\omega(x)+\omega(y)\leq R$ then we can exploit the classical span-contraction methodology for the bounded case; see e.g. Stettner (1999). Indeed, following the proof of (Pitera & Stettner, 2016, Lemma 3) it is enough to show that

[TABLE]

where $\bar{C}_{R}:=\{(x,y)\in E\times E:\omega(x)+\omega(y)\leq R\}$, and consider any

[TABLE]

for some fixed $\beta\in(0,1)$ satisfying

[TABLE]

The proof of (3.30) is based on contradiction. Assume there exists a sequence

[TABLE]

where $(x_{n},y_{n})\in\bar{C}_{R}$ , $f_{n},g_{n}\in\mathcal{C}_{\omega}(E)$, and $A_{n}\in\mathcal{B}(E)$ are such that $\|f_{n}\|_{\omega\textrm{-span}}\leq M$, $\|g_{n}\|_{\omega\textrm{-span}}\leq M$, and $\mathbb{H}^{f_{n},g_{n}}_{x_{n},y_{n}}(A_{n})\rightarrow 1$ (as $n\to\infty$). Following the proof of (Pitera & Stettner, 2016, Lemma 3) for any $x\in E$, $a\in\bar{U}$, $g\in\mathcal{C}_{\omega}(E)$ and $A\in\mathcal{B}(E)$, such that $\omega(x)\leq R$ and $\|f\|_{\omega\textrm{-span}}\leq M$, we get

[TABLE]

where $Z_{2}:=-\gamma\|f\|_{\omega}\int_{0}^{\delta}\omega(X_{s})\,ds+M\omega(X_{\delta})$. Using similar reasoning as in (3.35) and recalling ((A.3)) we get

[TABLE]

where $K=M-\delta\gamma\|f\|_{\omega}$ and $D\in\mathbb{R}$ is some fixed constant. Consequently, we get

[TABLE]

Thus, combining (3.33) with the fact that $\mathbb{H}^{f_{n},g_{n}}_{x_{n},y_{n}}(A_{n})\rightarrow 1$ we get

[TABLE]

On the other hand, from ((A.4)), for any $n\in\mathbb{N}$ and $(x_{n},y_{n})\in\bar{C}_{R}$, we get

[TABLE]

which leads to contradiction.

Combining steps 1), 2), and 3), we conclude the proof. ∎

Next, we show that the iterated sequence $(T^{n}_{\gamma}0)_{n=1}^{\infty}$ is bounded in $\omega$-span semi-norm.

Proposition 3.3**.**

For any $\gamma<0$ there exists $M\in\mathbb{R}_{+}$ such that

[TABLE]

Proof.

Let $\gamma<0$. For brevity, we use the notation $g_{n}:=R^{n}_{\gamma}0$ with the convention $g_{0}\equiv 0$. Moreover, we define $x_{n}^{*}:=\operatorname*{arg\,max}_{x\in U}g_{n}(x)$ and $Z:=\int_{0}^{\delta}f(X_{s})\,\operatorname{d}\!s$. Then, for any $n\in\mathbb{N}$ and $\beta>0$ we get

[TABLE]

where

[TABLE]

note that in (3.35) we used the following shift strategy: if a shift is applied to the process starting in $x$ then the same shift is applied to the process starting in $y$ with $K^{1}_{\beta}$ corresponding the the upper value bound; if no shift is applied to the process starting in $x$ then the shift to $x_{n}^{*}$ is applied to the process starting in $y$. Using ((A.2)), for any $\xi\in U$ and $y\in E$ we get $c(x,\xi)<c_{0}$ and $c(y,\xi)\geq-\|\hat{c}\|_{\beta,\omega}(1+\beta\omega(y))$. Consequently, for any $\beta>0$ we have $K^{1}_{\beta}<\infty$ and we can rewrite (3.35) as

[TABLE]

Noting that for any $x\in E$ we have

[TABLE]

we get

[TABLE]

Applying Hölder’s inequality for entropic utility measure with $p=2$ (see Lemma 6.1) we know that

[TABLE]

where, due to ((A.3)),

[TABLE]

Thus, setting

[TABLE]

and introducing $c_{n}:=\inf_{c\in\mathbb{R}}\|g_{n}+c\|_{\beta,\omega}$ we can rewrite (3.36) as

[TABLE]

where

[TABLE]

Next, using the fact that entropic risk measure is increasing with respect to the risk-averse parameter $\gamma$, noting that $\|g_{n}+c_{n}\|_{\beta,\omega}=\|g_{n}\|_{\beta,\omega\textrm{-span}}$, and using assumptions ((A.1))–((A.3)), we get

[TABLE]

Now, let us fix $\beta:=(2\sup_{\xi\in U}\omega(\xi))^{-1}$. Then, we get

[TABLE]

Consequently, there exists $R>0$ such that for any $n\in\mathbb{N}$ and $x\in E$ satisfying $\omega(x)>R$ we get

[TABLE]

Next, we show that there exist a constant $K^{\beta}_{3}>0$ such that for any $n\in\mathbb{N}$ and $x\in C_{R}$, where $C_{R}=\{x\in E\colon\omega(x)\leq R\}$, we get

[TABLE]

Using assumption ((A.4)) we know that there exists $\epsilon>0$ such that for any $n\in\mathbb{N}$ and $x\in C_{R}$ we get $\mathbb{P}_{x}\left[X_{\delta}\in U\right]>\epsilon$. Moreover, noting that for any $y\in U$ we have $g_{n}(y)\leq g_{n}(x_{n}^{*})$ and that the entropic utility measure is concave for $\gamma<0$ (which implies $a\mu_{x}^{\gamma}(\cdot)\leq\mu^{\gamma}_{x}(a\cdot)$ for $a\in(0,1)$), for any $n\in\mathbb{N}$ and $x\in C_{R}$ we get

[TABLE]

Let $Z_{x}:=\int_{0}^{\delta}(\omega(X_{s})-\omega(x))\operatorname{d}\!s$. Due to assumption ((A.3)), we know that

[TABLE]

Thus, we know that there exists $N\in\mathbb{R}$ such that

[TABLE]

Combining (3) with (3.42), for any $x\in C_{R}$ we get

[TABLE]

Consequently, setting $K_{3}^{\beta}:=\tfrac{1}{\gamma}\ln\tfrac{\epsilon}{2}+\|f\|_{\beta,\omega}(\delta+\beta N)$ we conclude the proof of (3.40).

Next, combining (3.39) and (3.40) we know that for any $n\in\mathbb{N}$ and $x\in E$ we get

[TABLE]

for constant parameters $a<1$ and $K^{4}_{\beta}\in\mathbb{R}_{+}$. Consequently, we can rewrite (3.38) as

[TABLE]

Using the standard geometric convergence arguments we know that (3.44) implies existence of a constant $M_{\beta}\in\mathbb{R}_{+}$ such that for any $n\in\mathbb{N}$ we get

[TABLE]

Finally, the equivalence of semi-norms $\|\cdot\|_{\beta,\omega\textrm{-span}}$ and $\|\cdot\|_{\omega\textrm{-span}}$ combined with the property

[TABLE]

concludes the proof. ∎

Combining Theorem 3.2 with Proposition 3.3, and using Banach’s fixed point theorem, we get the solution to Bellman equation (3.1); see Proposition 3.4. For brevity, we omit the proof; see second part of the proof in (Pitera & Stettner, 2016, Proposition 4) for details. Note that due to Proposition 3.3 we get solution to Bellman equation for any predefined $\gamma<0$. In particular, in contrast to (Pitera & Stettner, 2016, Proposition 4), we do not require $\gamma$ to be close to [math].

Proposition 3.4**.**

Let $\gamma<0$. Under assumptions ((A.1))–((A.4)) there exist a unique (up to an additive constant) $w^{\gamma}_{\delta}\in\mathcal{C}_{\omega}(E)$ and $\lambda^{\gamma}_{\delta}\in\mathbb{R}$, the solutions to Bellman equation (3.1).

4 Solution to the dyadic optimal control problem

Before we link the Bellman’s equation to the corresponding dyadic optimal control problem (2.5), let us show some supplementary results

Proposition 4.1**.**

The mapping $\gamma\to\lambda^{\gamma}_{\delta}$ is continuous on $(-\infty,0)$.

Proof.

Let us fix $a\in E$, and for any $\gamma<0$ set

[TABLE]

Note that $\bar{w}^{\gamma}_{\delta}$ is also a solution to Bellman equation (3.1), and from Proposition 3.3 we get $\|\gamma\bar{w}^{\gamma}_{\delta}\|_{\omega\textrm{-span}}\leq M$, where $M\in\mathbb{R}_{+}$ is a fixed constant. Moreover, since constant $M$ in Proposition 3.3 can be chosen uniformly on any compact subset of negative $\gamma$s, say $G$, for any $x\in E$, $m\in\mathbb{N}$, and $\gamma\in G$, using Theorem 3.2, we get

[TABLE]

Let us fix $x\in E$. By Proposition 3.1, the mappings $\gamma\to T_{\gamma}^{m}0(x)$ and $\gamma\to T_{\gamma}^{m}0(a)$ are continuous for any $m\in\mathbb{N}$. Therefore, using (4.1), for any $\gamma<0$, $m\in\mathbb{N}$, and a sequence $(\gamma_{n})_{n\in\mathbb{N}}$, such that $\gamma_{n}\to\gamma$, as $n\to\infty$, we get

[TABLE]

For any $\epsilon>0$ we can choose $m_{\epsilon}\in\mathbb{N}$, such that $c_{m_{\epsilon}}\leq\epsilon$. Consequently, letting $n\to\infty$ with a fixed $m_{\epsilon}$, we get $\limsup_{n\to\infty}|\gamma_{n}\bar{w}^{\gamma_{n}}_{\delta}(x)-\gamma\bar{w}^{\gamma}_{\delta}(x)|\leq\epsilon$. As the choice of $\epsilon$ is arbitrary, we get continuity of the mapping $\gamma\to\gamma\bar{w}^{\gamma}_{\delta}(x)$. Next, following the proof of Proposition 3.1, we see that the mapping $\gamma\to T_{\gamma}\gamma\bar{w}^{\gamma}_{\delta}(x)$ is also continuous. Consequently, noting that

[TABLE]

and using similar arguments as in (Pitera & Stettner, 2016, Proposition 4.8), we obtain continuity of $\gamma\to\gamma\lambda^{\gamma}_{\delta}$ on $(-\infty,0)$. This implies continuity of $\gamma\to\lambda^{\gamma}_{\delta}$ on $(-\infty,0)$, and completes the proof. ∎

Proposition 4.2**.**

For any $\gamma\in\mathbb{R}$ and $x\in E$, we get

[TABLE]

Proof.

Let us fix $\gamma\in\mathbb{R}$. Let $b_{2}\colon E\times E\to\mathbb{R}_{+}$ be given by

[TABLE]

In particular, note that for any $x\in E$ we get $\omega(X_{\delta})\leq b_{1}\omega(x)+b_{2}(x,X_{\delta}-x)$ and

[TABLE]

For completeness, let us outline the proof of (4.4). On the first hand, note that for any sequence $(x_{n})_{n\in\mathbb{N}}$, where $x_{n}\in E$, taking the limit $n\to\infty$, we get $\mu_{x_{n}}^{|\gamma|}\left(\omega(X_{\delta})-b_{1}\omega(x_{n})\right)\to\infty$ if and only if $\mathbb{E}_{x_{n}}\left[e^{|\gamma|(\omega(X_{\delta})-b_{1}\omega(x_{n}))}\right]\to\infty$. Consequently, since function $z\mapsto e^{z}$ is bounded from below, we get $\mu_{x_{n}}^{|\gamma|}\left(\omega(X_{\delta})-b_{1}\omega(x_{n})\right)\to\infty$ if and only if $\mu_{x_{n}}^{|\gamma|}\left(b_{2}(x_{n},X_{\delta}-x_{n})\right)\to\infty$. On the other hand, using ((A.3)), we get $\mu_{x_{n}}^{|\gamma|}\left(\omega(X_{\delta})-b_{1}\omega(x_{n})\right)\leq M_{2}(|\gamma|)$. These two facts imply (4.4).

Now, we fix $x\in E$ and introduce some additional auxiliary notation. Let $a:=\sup_{x\in U}\omega(x)$ and for any $n\in\mathbb{N}$ let

[TABLE]

where $X_{t}^{-}$ is the state of $(X_{t})$ before the (optional) shift; note that $\mathbbm{1}_{A_{n}}X_{n\delta}=\mathbbm{1}_{A_{n}}X^{-}_{n\delta}$ for $n\in\mathbb{N}$. For brevity, we also use $\mu^{\gamma}_{(x,V)}(\cdot\;|\;\mathcal{F}_{i})$ to denote the $\mathcal{F}_{i}$-conditional equivalent of $\mu^{\gamma}_{(x,V)}$.

Let us fix $V\in\mathbb{V}_{\delta}$ and $t\in\mathbb{T}_{\delta}$. Noting that $t=n\delta$ for some $n\in\mathbb{N}$, using monotonicity of $\mu^{\gamma}_{(x,v)}$, and ((A.3)), we get

[TABLE]

Using strong time-consistency and additivity of entropic utility, we have

[TABLE]

while from strong Markov property and (4.4) we get

[TABLE]

Consequently, combining (4.7), (4.6), and (4.5) we get

[TABLE]

Using similar reasoning recursively and noting that for $i=1,\ldots,n-1$ we have

[TABLE]

we finally get

[TABLE]

As the choice of $V\in\mathbb{V}_{\delta}$ and $t\in\mathbb{T}_{\delta}$ was arbitrary, and the upper bound in (4.9) is independent of both, we know that (4.2) is satisfied on $\mathbb{T}_{\delta}$ which concludes the proof. ∎

Finally, we are ready to link Bellman’s equation to the corresponding dyadic optimal control problem (2.5).

Proposition 4.3**.**

Under assumptions ((A.1))–((A.4)) we get

[TABLE]

i.e. the optimal value in problem (2.5) corresponds to the solution of Bellman equation (3.1).

Proof.

Proposition 4.2. For brevity and with slight abuse of notation, for any $n\in\mathbb{N}$ we set $T_{n}:=n\delta$ and

[TABLE]

note the exact dynamics of $Z_{n}$ is determined by an underlying strategy $V=\{(\tau_{i},\xi_{i})\}_{i=1}^{\infty}$.

First, let us show that

[TABLE]

Fix $n\in\mathbb{N}$, $p>1$, and set $\bar{\gamma}:=p\gamma$. Let $q$ be the conjugate index for $p$ and let $\phi:=-\|w^{\bar{\gamma}}_{\delta}\|_{\omega}q\gamma$. For the strategy $\hat{V}=\{(\hat{\tau}_{i},\hat{\xi}_{i})\}_{i=1}^{\infty}\in\mathbb{V}_{\delta}$ determined by the Bellman equation (3.1) for $\bar{\gamma}$, using reverse Hölder’s inequality for $p$ and $q$ (see Lemma 6.1), we get

[TABLE]

Using Proposition 4.2 we know that $\sup_{n\in\mathbb{N}}\mu_{(x,V)}^{\phi}\left(\omega(X_{T_{n}})\right)<\infty$. Consequently, letting $n\to\infty$ we obtain

[TABLE]

Now, recall that $\bar{\gamma}=p\gamma$ and note that (4.12) holds for any choice of $p>1$. Thus, using Proposition 4.1 and letting $p\to 1$, we get that $\lambda_{\delta}^{p\gamma}\to\lambda_{\delta}^{\gamma}$. This concludes the proof of (4.10).

Second, we prove inequality

[TABLE]

Again, we fix $n\in\mathbb{N}$ and $p>1$. Let $\bar{\gamma}:=\gamma/p$ and $\phi:=-\|w^{\bar{\gamma}}_{\delta}\|_{\omega}q\bar{\gamma}$, where $q$ is the conjugate index for $p$. For any strategy $V\in\mathbb{V}_{\delta}$, using Hölder’s inequality for $p$ and $q$ (see Lemma 6.1), we get

[TABLE]

As before, using Proposition 4.2 and letting $n\to\infty$, for any $V\in\mathbb{V}_{\delta}$ we obtain

[TABLE]

As the choice of $V\in\mathbb{V}$ is arbitrary we get

[TABLE]

Finally, as in the proof of (4.10), using Proposition 4.1 and letting $p\to 1$, we get $\lambda_{\delta}^{\gamma/p}\to\lambda_{\delta}^{\gamma}$, which concludes the proof of (4.13), and Proposition 4.3.

∎

Remark 4.4* (Application of entropic Hölder’s inequalities).*

The key step in the proof of Proposition 4.3 is the application on the Holder’s inequality and reverse Holder’s inequality for the entropic risk; see Lemma 6.1. Using the induced superadditivity and subadditivity property (for different risk averse parameters), one can split the main dynamics from $w^{\gamma}_{\delta}(\cdot)$. It is interesting to note that the same approach could be applied in (Pitera & Stettner, 2016, Proposition 5), i.e. using our framework it is easy to show that the solution to the Bellman’s equation is the optimal solution, without imposing any additional constraints as in (Pitera & Stettner, 2016, Proposition 5).

Remark 4.5* (Full time-grid).*

While in Proposition 4.2 and Proposition 4.3 we restricted ourselves to the dyadic time-grid, the results holds (under additional mild assumptions) on the full-time grid, i.e. with objective function (2.1) replaced by

[TABLE]

Following comments from Remark 2.1 and treating $b_{1}$ and $M_{i}$ in ((A.3)) as functions of $\delta$, let us assume that $M_{i}(\gamma,\delta)\to 0$ as $\delta\to 0$, for any $\gamma\in\mathbb{R}$. For brevity, let us only outline how to extend the proof of Proposition 4.2. Let $t>0$ be such that $t\not\in\mathbb{T}_{\delta}$ and let $V\in\mathbb{V}_{\delta}$. We know that there exists $\delta_{0}<\delta$ such that $M:=\sup_{\delta\in(0,\delta_{0}]}M_{i}(|\gamma|,\delta)<\infty$. Also, we know that there exist $n\in\mathbb{N}$ and $m\in\mathbb{N}$ such that $t=n\delta+m\delta_{0}+\epsilon$, where $m\delta_{0}<\delta$ and $\epsilon\in[0,\delta_{0})$. For brevity we set $t_{0}:=n\delta+\epsilon$. Using ((A.3)) $m$-times for time step $\delta_{0}$ and once for time step $\epsilon$ (if required), and using notation introduced in (4.3), we get

[TABLE]

Now, using similar arguments as in the proof of (4.9), we get

[TABLE]

where $\tilde{M}$ and $\tilde{M}_{2}(|\gamma|,\delta)$ is constructed as in (4.4). As the choice of $\delta_{0}$ was independent of the choice of $t$ and $V$, so is the upper bound in (4.15). This concludes the proof of (4.2) for $t\in\mathbb{T}$.

5 Reference examples

In this section we want to show examples of processes satisfying assumptions ((A.1))–((A.4)). For brevity, as assumptions ((A.1)), ((A.2)), and ((A.4)) are rather standard, we decided to focus on assumption ((A.3)) and describe only the dynamic of the uncontrolled process; one could easily enhance this process to get a proper example satisfying ((A.1))–((A.4)). Example 5.1 focus on Ito-like diffusion process, Example 5.2 is linked to regular step processes studied in Blumenthal & Getoor (2007), and Example 5.3 considers a piecewise deterministic process introduced in Davis (1984) and studied later in the context of control theory in Bäuerle & Rieder (2011). For simplicity, in the first two examples we assume that $E=\mathbb{R}^{d}$ and $\delta<1$, and in the third we set $E=\mathbb{R}$.

Example 5.1** (Ito-like diffusion).**

Let $(X_{t})$ be a solution to equation

[TABLE]

where matrix $A\in\mathbb{R}^{d\times d}$ is stable (real parts of its eigenvalues are negative) and diagonalizable (its geometric and algebraic multiplicities coincides), functions $g:\mathbb{R}^{d}\to\mathbb{R}^{d}$ and $\sigma:\mathbb{R}^{d}\to\mathbb{R}^{d\times d}$ are bounded, and $(W_{t})$ is $\mathbb{R}^{d}$-valued Brownian motion. Additionally, we assume that $\sigma$ is Lipschitz continuous to guarantee strong solution of (5.1) with $g\equiv 0$. Then, there exists a weak solution to (5.1) given by

[TABLE]

Let $\omega(x):=\max_{i\in\{1,\ldots,d\}}|x_{i}|$ for $x\in\mathbb{R}^{d}$. Then, for any $t\leq 1$ and $\gamma\in\mathbb{R}$ we get

[TABLE]

where $\alpha\in\mathbb{R}_{+}$ is a (negative of) maximal real part of eigenvalues of $A$ and $\|\cdot\|_{\infty}$ denotes the supremum norm. We now show that the last term in (5.3) could be uniformly bounded for any $\gamma\in\mathbb{R}$. For simplicity, and without loss of generality, we assume that $\gamma>0$; recall that entropic risk measure is monotone with respect to $\gamma$. Let

[TABLE]

and let $Z_{i}(t)$ denote the i-th component of $Z(t)$, for $i=1,\ldots,d$. Notice that for any $\gamma>0$ and $x\in E$ we have

[TABLE]

By the local martingale property of $e^{\gamma\sum_{i=1}^{d}(-1)^{s_{i}}Z_{i}(t)-{1\over 2}\gamma^{2}\langle\sum_{i=1}^{d}(-1)^{s_{i}}Z_{i}\rangle_{t}}$ (we refer to Problem 3.38 in Karatzas & Shreve (1998)), for any $x\in E$ we get

[TABLE]

This completes the proof the second estimate in ((A.3)). The first estimate in ((A.3)), i.e. inequality $\mu_{x}^{\gamma}(\int_{0}^{t}\omega(X_{s})ds)\leq\omega(x)+M_{1}(\gamma)$, can be obtained in a similar way by exploiting property (5.2).

Example 5.2** (Regular step process).**

Let $(X_{t})$ be a regular step process that is constructed using the following logic: a particle is starting from point $X_{0}=z_{0}$ and remains in there for exponentially distributed time with parameter $r(z_{0})$. Then, it jumps to another (randomly chosen) state $z_{1}$ and remains there for exponentially distributed time with parameter $r(z_{1})$, and so on. The intensity function $r\colon\mathbb{R}^{d}\to\mathbb{R}_{+}$ is given by $r(\cdot):=\max\left\{\|\cdot\|^{1+\epsilon},r_{0}\right\}$, where $r_{0}>0$ and $\epsilon>0$ are fixed constants, and where $\|\cdot\|$ is the standard $\mathbb{R}^{d}$-norm. The jump from $z_{n}\in\mathbb{R}^{d}$ to $z_{n+1}\in\mathbb{R}^{d}$ (for $n\in\mathbb{N}$) is made according to the transition law $Q(z_{n},\cdot)$ such that

[TABLE]

where $(w_{n})$ is an i.i.d. sequence of bounded $\mathbb{R}^{d}$-valued random vectors, and the function $A\colon\mathbb{R}^{d}\to\mathbb{R}^{d}$ satisfy $\lim_{\|x\|\to\infty}\tfrac{1}{\|x\|}\|A(x)\|<1$. Then, there exists a constant $K\in\mathbb{R}_{+}$ and $\beta\in(0,1)$ such that $\|A(x)\|+\|w_{i}\|\leq\beta\|x\|+K$ for $x\in E$ and $i\in\mathbb{N}$. Consequently, for any $n\in\mathbb{N}$, we get $\|z_{n+1}\|\leq\beta\|z_{n}\|+K$ and, by iteration,

[TABLE]

Let $\omega(\cdot):=\|\cdot\|$ and let $\tau(x)$ denote the time of the first process jump for any fixed starting point $x\in E$. Then, for any $x\in E$ and $\gamma>0$, using (5.8), we get

[TABLE]

Now, set $R(\gamma):=\max\{\sqrt[\epsilon]{\gamma/\delta},r_{0}\}$ and recall that $C_{R(\gamma)}=\{x\in\mathbb{R}^{d}:\omega(x)<R(\gamma)\}$. For any $x\in C_{R(\gamma)}$, we get $e^{-\delta r(x)+\gamma\omega(x)}\leq e^{\gamma R(\gamma)}$, while for $x\not\in C_{R}$ we have

[TABLE]

Consequently, from (5.2), we get

[TABLE]

where $\tilde{K}(\gamma)$ is some fixed constant independent of $x\in E$. This completes the proof of the second inequality in ((A.3)), as (5.11) could be rewritten as $\mu^{\gamma}_{x}(\omega(X_{\delta}))\leq\beta\omega(x)+\tilde{K}(\gamma)$, for $x\in E$. The proof of the first inequality in ((A.3)) follow directly from (5.8).

Example 5.3** (Piecewise deterministic process).**

Assume that $(X_{t})$ is a piecewise deterministic process. The deterministic part is a solution to a stable differential equation

[TABLE]

with initial state $X_{0}=x$. The process follows this dynamics till (random) jump moment, and then is subject to immediate shift after which its evolution follows the same deterministic logic till next jump occurs, and so on. We assume that the sequence of jumps, say $(\tau_{n})$, is such that $(\tau_{n+1}-\tau_{n})$ is i.i.d. and exponentially distributed with fixed intensity $r>0$. The shifts are made according to transition measure such that

[TABLE]

where $w_{n}$ is a sequence of i.i.d. standard normal random variables and function $A\colon\mathbb{R}\to\mathbb{R}$ satisfy $|A(x)|\leq|x|+K$, for $K>0$. Assuming suitable regularity of $F$, for any $t<\tau_{1}$ and initial state $x$, we get $X_{t}=\phi(x,t)$, where $\phi$ is a continuous function. Moreover, we assume that $\phi$ is such that for any $x\in E$ we get $|\phi(x,t)|\leq e^{-\alpha t}|x|+M$, where $\alpha,M>0$ are some predefined constants that are independent of $x$. Then, we get

[TABLE]

and, for any $n\in\mathbb{N}$, by induction,

[TABLE]

Consequently, for any $\gamma>0$, setting $\omega(\cdot):=\|\cdot\|$, $\beta:=e^{-\alpha\delta}$, $\tau_{0}:=0$, $w_{0}:=0$, and $D(\gamma):=\tfrac{1}{\gamma}\ln\mathbb{E}[e^{\gamma|w_{1}|}]$, and noting that $(w_{i})$ is independent of $(\tau_{i})$, we get

[TABLE]

Next, noting that

[TABLE]

we can rewrite (5.14) as

[TABLE]

where $\tilde{D}(\gamma)$ is some constant that is independent of $x$; this concludes the proof of the left inequality in ((A.3)). The second inequality in ((A.3)) follows in a similar manner.

6 Appendix

For simplicity, in this section we assume that a probability space is fixed and for any $\gamma\in\mathbb{R}\setminus\{0\}$ and $X\in L^{0}$ we set

[TABLE]

Lemma 6.1** (Hölder’s inequalities for entropic utility measure).**

Let $\gamma<0$ (resp. $\gamma>0$). Then, for any $p>1$ and the corresponding conjugate index $q$ we get

[TABLE]

where $X,Y\in L^{0}$.

Proof.

We only show proof for $\gamma<0$ as the proof for $\gamma>0$ is analogous. Let us fix $p>1$. Using Hölder’s inequality applied to $e^{\gamma X}$ and $e^{\gamma Y}$ we get

[TABLE]

Taking logarithm on both sides and multiplying by $1/\gamma<0$ we get

[TABLE]

which is equivalent to (6.1). Next, applying (6.1) to $\tilde{\gamma}=\gamma/p$, $\tilde{X}:=X+Y$, and $\tilde{Y}:=-Y$, we get

[TABLE]

from which (6.2) follows. ∎