A Finite Horizon Optimal Switching Problem with Memory and Application to Controlled SDDEs

TL;DR

This paper addresses a finite horizon optimal switching problem with memory, establishing existence of optimal controls, and applies it to controlled stochastic delay differential equations, including practical hydro-power revenue maximization.

Contribution

It introduces a probabilistic approach to prove existence of optimal controls in switching problems with memory and applies it to stochastic delay differential equations with real-world relevance.

Findings

Existence of optimal control established using Snell envelopes.

Application to impulse control problems for SDDEs with jump processes.

Relevance demonstrated in hydro-power revenue maximization.

Abstract

We consider an optimal switching problem where the terminal reward depends on the entire control trajectory. We show existence of an optimal control by applying a probabilistic technique based on the concept of Snell envelopes. We then apply this result to solve an impulse control problem for stochastic delay differential equations driven by a Brownian motion and an independent compound Poisson process. Furthermore, we show that the studied problem arises naturally when maximizing the revenue from operation of a group of hydro-power plants with hydrological coupling.

Equations

$$\mathbb{E}\bigg{[}\int_{0}^{T}\phi_{\xi_{s}}(s)ds+\psi_{\xi_{T}}-\sum_{j=1}^{N}c_{\beta_{j-1},\beta_{j}}(\tau_{j})\bigg{]},$$

$$J(u)$$

$$\Psi ({\tau }_1,\dots ,{\tau }_N;{\beta }_1,\dots ,{\beta }_N)$$

$$c_{b_1,b_2}(t_1)+\cdots +c_{b_n,b_1}(t_n)\ge ϵ,$$

$$J(u^{\ast })=\underset{u\in \mathcal{U}}{\sup }J(u).$$

$$J(\hat{u})$$

$$Z_{\gamma}=\mathop{\rm{ess}\,\sup}_{\tau\in\mathcal{T}_{\gamma}}\mathbb{E}\left[U_{\tau}\big{|}\mathcal{F}_{\gamma}\right].$$

$$Z_t=M_t-K_t^c-K_t^d.$$

$$Z_{\theta}=\mathbb{E}\left[U_{\tau^{*}_{\theta}}\big{|}\mathcal{F}_{\theta}\right].$$

$$V_s^{t;b,u}$$

$$\displaystyle\lim_{l\to\infty}\sup_{u\in\mathcal{U}}\mathbb{E}\Big{[}$$

$$\displaystyle\lim_{l\to\infty}\sup_{u\in\mathcal{U}_{\Gamma^{l}(\eta_{n})}}\mathbb{E}\Big{[}$$

$$Y_s^{t;b}$$

$$\lim_{l\to\infty}\mathbb{E}\big{[}\sup_{s\in[0,T]}|Y^{\Gamma^{l}(\eta);\mathbb{b}}_{s}-Y^{\eta;\mathbb{b}}_{s}|^{2}\big{]}=0$$

$$\lim_{l\to\infty}\mathbb{E}\big{[}\sup_{s\in[0,T]}|Y^{\Gamma^{l}(\eta),s\vee\Gamma^{l}(\eta_{n});\mathbb{b},b}_{s}-Y^{\eta,s\vee\eta_{n};\mathbb{b},b}_{s}|^{2}\big{]}=0.$$

$$\displaystyle\tau^{*}_{j}:=\inf\Big{\{}$$

$$\beta^{*}_{j}\in\mathop{\arg\max}_{\beta\in\mathcal{I}^{-\beta^{*}_{j-1}}}\Big{\{}-c_{\beta_{j-1}^{*},\beta}(\tau_{j}^{*})+Y^{\tau^{*}_{1},\ldots,\tau^{*}_{j};\beta^{*}_{1},\ldots,\beta^{*}_{j-1},\beta}_{\tau^{*}_{j}}\Big{\}}$$

$$\displaystyle Y^{\tau^{*}_{1},\ldots,\tau^{*}_{j};\beta^{*}_{1},\ldots,\beta^{*}_{j}}_{s}=\mathop{\rm{ess}\,\sup}_{\tau\in\mathcal{T}_{s}}\mathbb{E}\Big{[}\mathbbm{1}_{[\tau\geq T]}\Psi(\tau^{*}_{1},\ldots,\tau^{*}_{j};\beta^{*}_{1},\ldots,\beta^{*}_{j})$$

$$\displaystyle\qquad+\mathbbm{1}_{[\tau<T]}\max_{\beta\in\mathcal{I}^{-\beta^{*}_{j}}}\left\{-c_{\beta^{*}_{j},\beta}(\tau)+Y^{\tau^{*}_{1},\ldots,\tau^{*}_{j},\tau;\beta^{*}_{1},\ldots,\beta^{*}_{j},\beta}_{\tau}\right\}\Big{|}\mathcal{F}_{s}\Big{]}$$

$$\displaystyle=\mathbb{E}\Big{[}\mathbbm{1}_{[\tau^{*}_{j+1}\geq T]}\Psi(\tau^{*}_{1},\ldots,\tau^{*}_{j};\beta^{*}_{1},\ldots,\beta^{*}_{j})$$

$$\displaystyle\qquad+\mathbbm{1}_{[\tau^{*}_{j+1}<T]}\left\{-c_{\beta^{*}_{j},\beta^{*}_{j+1}}(\tau^{*}_{j+1})+Y^{\tau^{*}_{1},\ldots,\tau^{*}_{j+1};\beta^{*}_{1},\ldots,\beta^{*}_{j+1}}_{\tau^{*}_{j+1}}\right\}\Big{|}\mathcal{F}_{s}\Big{]},$$

$$\displaystyle Y^{\mathbb{t};\mathbb{b}}_{s}=\mathop{\rm{ess}\,\sup}_{u\in\mathcal{U}_{s\vee t_{n}}}\mathbb{E}\Big{[}\Psi(\mathbb{t},\tau_{1},\ldots,\tau_{N};\mathbb{b},\beta_{1},\ldots,\beta_{N})-\sum_{j=1}^{N}c_{\beta_{j-1},\beta_{j}}(\tau_{j})\Big{|}\mathcal{F}_{s}\Big{]},$$

$$\displaystyle U^{\mathbb{t};\mathbb{b}}:=\Big{(}$$

$$\displaystyle Y^{\mathbb{t};\mathbb{b}}_{\theta}=\mathbb{E}\Big{[}\mathbbm{1}_{[\gamma_{\theta}=T]}\Psi(\mathbb{t};\mathbb{b})+\mathbbm{1}_{[\gamma_{\theta}<T]}\max_{\beta\in\mathcal{I}^{-b_{n}}}\left\{-c_{b_{n},\beta}(\gamma_{\theta})+Y^{\mathbb{t},\gamma_{\theta};\mathbb{b},\beta}_{\gamma_{\theta}}\right\}\Big{|}\mathcal{F}_{\theta}\Big{]}.$$

$$\displaystyle\Big{(}\mathbbm{1}_{[s=T]}\Psi_{0}+\mathbbm{1}_{[s<T]}\max_{\beta\in\mathcal{I}^{-b_{0}}}\left\{-c_{b_{0},\beta}(s)+Y^{s,\beta}_{s}\right\}:\>0\leq s\leq T\Big{)},$$

$$Y_0$$

$$\displaystyle=\mathbb{E}\Big{[}\mathbbm{1}_{[\tau^{*}_{1}=T]}\Psi_{0}+\mathbbm{1}_{[\tau^{*}_{1}<T]}\max_{\beta\in\mathcal{I}^{-b_{0}}}\left\{-c_{b_{0},\beta}(\tau^{*}_{1})+Y^{\tau^{*}_{1},\beta}_{\tau^{*}_{1}}\right\}\Big{]}$$

$$\displaystyle=\mathbb{E}\Big{[}\mathbbm{1}_{[\tau^{*}_{1}=T]}\Psi_{0}+\mathbbm{1}_{[\tau^{*}_{1}<T]}\left\{-c_{b_{0},\beta^{*}_{1}}(\tau^{*}_{1})+Y^{\tau^{*}_{1},\beta^{*}_{1}}_{\tau^{*}_{1}}\right\}\Big{]}.$$

$${\hat{Y}}_s^M$$

$${\hat{U}}_s^M$$

$$\displaystyle\qquad\mathbbm{1}_{[s=T]}\Psi(z_{k_{1}},\ldots,z_{k_{j}};b_{1},\ldots,b_{j})+\mathbbm{1}_{[s<T]}\max_{\beta\in\mathcal{I}^{-b_{j}}}\Big{\{}-c_{b_{j},\beta}(s\vee z_{k_{j}})+Y^{z_{k_{1}},\ldots,z_{k_{j}},s\vee z_{k_{j}};b_{1},\ldots,b_{j},\beta}_{s}\Big{\}}\Big{)},$$

$$\text{mathbbm}{1}_{(z_{k_1-1},z_{k_1}]}({\tau }_1^{\ast })\cdots \text{mathbbm}{1}_{(z_{k_j-1},z_{k_j}]}({\tau }_j^{\ast })\text{mathbbm}{1}_{[{\beta }_1^{\ast }=b_1]}\cdots \text{mathbbm}{1}_{[{\beta }_j^{\ast }=b_j]}{Y}_s^{z_{k_1},\dots ,z_{k_j};b_1,\dots ,b_j},$$

$$\text{mathbbm}{1}_{(z_{k_1-1},z_{k_1}]}({\tau }_1^{\ast })\cdots \text{mathbbm}{1}_{(z_{k_j-1},z_{k_j}]}({\tau }_j^{\ast })\text{mathbbm}{1}_{[{\beta }_1^{\ast }=b_1]}\cdots \text{mathbbm}{1}_{[{\beta }_j^{\ast }=b_j]}{Z}_s$$

Full text

1 Introduction

The standard optimal switching problem (sometimes referred to as starting and stopping problem) is a stochastic optimal control problem of impulse type that arises when an operator controls a dynamical system by switching between the different members in a set of operation modes $\mathcal{I}=\{1,\ldots,m\}$. In the two-modes setting ($m=2$) the modes may represent, for example, “operating” and “closed” when maximizing the revenue from mineral extraction in a mine as in [6]. In the multi-modes setting the operating modes may represent different levels of power production in a power plant when the owner seeks to maximize her total revenue from producing electricity [7] or the states “operating” and “closed” of single units in a multi-unit production facility as in [5].

In optimal switching the control takes the form $u=(\tau_{1},\ldots,\tau_{N};\beta_{1},\ldots,\beta_{N})$, where $\tau_{1}\leq\tau_{2}\leq\cdots\leq\tau_{N}$ is a sequence of times when the operator intervenes on the system and $\beta_{j}\in\mathcal{I}^{-\beta_{j-1}}:=\mathcal{I}\setminus\{\beta_{j-1}\}$ is the mode in which the system is operated during $[\tau_{j},\tau_{j+1})$. The standard multi-modes optimal switching problem in finite horizon ($T<\infty$) can be formulated as finding the control that maximizes

[TABLE]

where $\xi_{t}=b_{0}\mathbbm{1}_{[0,\tau_{1})}(t)+\sum_{j=1}^{N}\beta_{j}\mathbbm{1}_{[\tau_{j},\tau_{j+1})}(t)$ is the operation mode (when starting in a predefined mode $b_{0}\in\mathcal{I}$), $\phi_{b}$ and $\psi_{b}$ are the running and terminal reward in mode $b\in\mathcal{I}$, respectively and $c_{b,b^{\prime}}(t)$ is the cost incurred by switching from mode $b$ to mode $b^{\prime}$ at time $t\in[0,T]$.

The standard optimal switching problem has been thoroughly investigated in the last decades after being popularised in [6]. In [16] a solution to the two-modes problem was found by rewriting the problem as an existence and uniqueness problem for a doubly reflected backward stochastic differential equation. In [11] existence of an optimal control for the multi-modes optimal switching problem was shown by a probabilistic method based on the concept of Snell envelopes. Furthermore, existence and uniqueness of viscosity solutions to the related Bellman equation was shown for the case when the switching costs are constant and the underlying uncertainty is modeled by a stochastic differential equation (SDE) driven by a Brownian motion. In [12] the existence and uniqueness results of viscosity solutions was extended to the case when the switching costs depend on the state variable. Since then, results have been extended to Knightian uncertainty [18, 17, 8] and non-Brownian filtration and signed switching costs [24]. For the case when the underlying uncertainty can be modeled by a diffusion process, generalization to the case when the control enters the drift and volatility term was treated in [14]. This was further developed to include state constraints in [20]. Another important generalization is to the case when the operator only has partial information about the present state of the diffusion process as treated in [23].

In the present work we consider the setting with running and terminal rewards that depend on the entire history of the control. We also show that a special case of the type of switching problems that we consider is that of a controlled stochastic delay differential equation (SDDE), driven by a finite intensity Lévy process.

To motivate our problem formulation we consider the situation when an operator of two hydro-power plants, located in the same river, wants to maximize her revenue from producing electricity during a fixed operation period. We assume that each plant has its own water reservoir. The power production in a hydropower plant depends on the drop height from the water level of the reservoir to the outlet and thus on the amount of water in the reservoir. As water that passes through the upstream plant will eventually reach the reservoir of the downstream plant we need to consider part of the control history in the upstream plant when optimizing operation of the downstream plant.

In this setting our cost functional can be written

[TABLE]

where $N_{s}:=\max\{j:\tau_{j}\leq s\}$. The contribution of the present work is twofold. First, we show that the problem of maximizing $J$ can be solved under certain assumptions on $\phi$, $\psi$ and the switching costs $c_{\cdot,\cdot}$ by finding an optimal control in terms of a family of interconnected value processes, that we refer to as a verification family. We then show that the revenue maximization problem of the hydro-power producer can be formulated as an impulse control problem where the uncertainty is modeled by a controlled SDDE and use our initial result to find an optimal control for this problem.

The remainder of the article is organized as follows. In the next section we state the problem, set the notation used throughout the article and detail the set of assumptions that are made. Then, in Section 3 a verification theorem is derived. This verification theorem is an extension of the original verification theorem for the multi-modes optimal switching problem developed in [11] and presumes the existence of a verification family. In Section 4 we show that, under the assumptions made, there exists a verification family, thus proving existence of an optimal control for the switching problem with cost functional $J$. In Section 5 we more carefully investigate the example of the hydro-power producer and show that the case of a controlled SDDE fits into the problem description investigated in Sections 3 and 4.

2 Preliminaries

We consider a finite horizon problem and thus assume that the terminal time $T$ is fixed with $T<\infty$.

We let $(\Omega,\mathcal{F},\mathbb{F},\mathbb{P})$ be a probability space, with $\mathbb{F}:=(\mathcal{F}_{t})_{0\leq t\leq T}$ a filtration satisfying the usual conditions in addition to being quasi-left continuous.

Remark 2.1**.**

Recall here the concept of quasi-left continuity: A càdlàg process $(X_{t}:0\leq t\leq T)$ is quasi-left continuous if for each predictable stopping time $\gamma$ and every announcing sequence of stopping times $\gamma_{k}\nearrow\gamma$ we have $X_{\gamma-}:=\lim\limits_{k\to\infty}X_{\gamma_{k}}=X_{\gamma}$, $\mathbb{P}$-a.s. A filtration is quasi-left continuous if $\mathcal{F}_{\gamma}=\mathcal{F}_{\gamma-}$ for every predictable stopping time $\gamma$.

Throughout we will use the following notation:

• •

$\mathcal{P}_{\mathbb{F}}$ is the $\sigma$-algebra of $\mathbb{F}$-progressively measurable subsets of $[0,T]\times\Omega$.

• •

For $p\geq 1$, we let $\mathcal{S}^{p}$ be the set of all $\mathbb{R}$-valued, $\mathcal{P}_{\mathbb{F}}$-measurable, càdlàg processes $(Z_{t}:0\leq t\leq T)$ such that, $\mathbb{P}$-a.s., $\mathbb{E}\left[\sup_{t\in[0,T]}|Z_{t}|^{p}\right]<\infty$ and let $\mathcal{S}_{\textit{qlc}}^{p}$ be the subset of processes that are quasi-left continuous.

• •

We let $\mathcal{T}$ be the set of all $\mathbb{F}$-stopping times and for each $\gamma\in\mathcal{T}$ we let $\mathcal{T}_{\gamma}$ be the corresponding subsets of stopping times $\tau$ such that $\tau\geq\gamma$, $\mathbb{P}$-a.s.

• •

We let $\mathcal{U}$ be the set of all $u=(\tau_{1},\ldots,\tau_{N};\beta_{1},\ldots,\beta_{N})$, where $(\tau_{j})_{j=1}^{N}$ is a non-decreasing sequence of $\mathbb{F}$-stopping times (such that $\lim_{j\to\infty}\tau_{j}=T$, $\mathbb{P}$-a.s.) and $\beta_{j}\in\mathcal{I}^{-\beta_{j-1}}$ is $\mathcal{F}_{\tau_{j}}$-measurable (with $\beta_{0}:=b_{0}$, the initial operation mode).

• •

We let $\mathcal{U}^{f}$ denote the subset of $u\in\mathcal{U}$ for which $N$ is finite $\mathbb{P}$-a.s. (*i.e. 

$\mathcal{U}^{f}:=\{u\in\mathcal{U}:\>\mathbb{P}\left[\{\omega\in\Omega:N(\omega)>k,\>\forall k>0\}\right]=0\}$*) and for all $k\geq 0$ we let $\mathcal{U}^{k}:=\{u\in\mathcal{U}:\>N\leq k\}$. For $\gamma\in\mathcal{T}$ we let $\mathcal{U}_{\gamma}$ (and $\mathcal{U}_{\gamma}^{f}$ resp. $\mathcal{U}_{\gamma}^{k}$) be the subset of $\mathcal{U}$ (and $\mathcal{U}^{f}$ resp. $\mathcal{U}^{k}$) with $\tau_{1}\in\mathcal{T}_{\gamma}$.

• •

We define the set $\mathcal{D}:=\{(t_{1},\ldots;b_{1},\ldots):t_{1}\leq t_{2}\leq\cdots,\>b_{j+1}\in\mathcal{I}^{-b_{j}}\}$ and let $\mathcal{D}^{f}$ be the corresponding subset of all finite sequences.

• •

For all $n\geq 0$, we let $\bar{\mathcal{I}}^{n}:=\{(b_{1},\ldots,b_{n})\in\mathcal{I}^{n}:\,b_{j}\in\mathcal{I}^{-b_{j-1}}\}$ and $\bar{\mathcal{T}}^{n}:=\{(\eta_{1},\ldots,\eta_{n})\in\mathcal{T}^{n}:\,\eta_{1}\leq\eta_{2}\leq\cdots\leq\eta_{n}\}$.

• •

For $l\geq 0$, we let $\Pi_{l}:=\{0,T2^{-l},2T2^{-l},\ldots,T\}$ and define the map $\Gamma^{l}:\cup_{j\geq 1}\bar{\mathcal{T}}^{j}\to\cup_{j\geq 1}\bar{\mathcal{T}}^{j}$ as $\Gamma^{l}(\eta_{1},\ldots,\eta_{j}):=(\inf\{s\in\Pi_{l}:\,s\geq\eta_{1}\},\ldots,\inf\{s\in\Pi_{l}:\,s\geq\eta_{j}\})$ for all $\eta\in\bar{\mathcal{T}}^{j}$.

To make notation more efficient we introduce the $\mathcal{F}_{T}$-measurable function:

[TABLE]

2.1 Problem formulation

In the above notation, our problem can be characterized by two objects:

• •

A $\mathcal{F}_{T}\otimes\mathcal{B}(\mathcal{D})$-measurable map $\Psi:\mathcal{D}\to\mathbb{R}$.

• •

A collection, $(c_{b,b^{\prime}}:\Omega\times[0,T]\to\mathbb{R})_{(b,b^{\prime})\in\bar{\mathcal{I}}^{2}}$, of $\mathcal{P}_{\mathbb{F}}$-measurable processes.

We will make the following preliminary assumptions on these objects:

Assumption 2.2**.**

1. (i)

The function $\Psi$ is $\mathbb{P}$-a.s. right-continuous in the intervention times and bounded in the sense that:

1. a)

$\sup_{u\in\mathcal{U}}\mathbb{E}[|\Psi(\tau_{1},\ldots;\beta_{1},\ldots)|^{2}]<\infty$. 2. b)

For all $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and any333Throughout we will use $t_{n}$ and $b_{n}$ to denote that last element in the vector $\mathbb{t}$ and $\mathbb{b}$, respectively, whenever $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$.* $b\in\mathcal{I}^{-b_{n}}$ we have*

$\sup_{u\in\mathcal{U}}\mathbb{E}[\sup_{s\in[t_{n},T]}|\Psi(\mathbb{t},s,\tau_{1}\vee s,\ldots;\mathbb{b},b,\beta_{1},\ldots)|^{2}]<\infty$. 2. (ii)

For each $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and any $b\in\mathcal{I}^{-b_{n}}$ we have $\Psi(\mathbb{t};\mathbb{b})>\Psi(\mathbb{t},T;\mathbb{b},b)-c_{b_{n},b}(T)$, $\mathbb{P}$-a.s. 3. (iii)

We assume that $(c_{b,b^{\prime}})_{(b,b^{\prime})\in\bar{\mathcal{I}}^{2}}\in(\mathcal{S}_{\textit{qlc}}^{2})^{m(m-1)}$ are such that:

1. a)

$c_{b,b^{\prime}}\geq 0$, $\mathbb{P}$-a.s. 2. b)

There is an $\epsilon>0$ such that for each $(t_{1},\ldots,t_{n},b_{1},\ldots,b_{n})$ with $0\leq t_{1}\leq\cdots\leq t_{n}\leq T$ and $b_{1}\in\mathcal{I}^{-b_{n}}$, and $b_{j}\in\mathcal{I}^{-b_{j-1}}$ for $j=2,\ldots,n$, we have

[TABLE]

$\mathbb{P}$-a.s.

The above assumptions are mainly standard assumptions for optimal switching problems translated to our setting. Assumptions (i.a) and (iii.a) together imply that the expected maximal reward is finite. Assumption (ii) implies that it is never optimal to switch at the terminal time. We show below that the “no-free-loop” condition (iii.b) together with (i.a) implies that, with probability one, the optimal control (whenever it exists) can only make a finite number of switches.

We consider the following problem:

Problem 1. Find $u^{*}\in\mathcal{U}$, such that

[TABLE]

∎

As a step in solving Problem 1 we need the following proposition which is a standard result for optimal switching problems and is due to the “no-free-loop” condition.

Proposition 2.3**.**

Suppose that there is a $u^{*}\in\mathcal{U}$ such that $J(u^{*})\geq J(u)$ for all $u\in\mathcal{U}$. Then $u^{*}\in\mathcal{U}^{f}$.

Proof. Pick $\hat{u}:=(\hat{\tau}_{1},\ldots,\hat{\tau}_{\hat{N}};\hat{\beta}_{1},\ldots,\hat{\beta}_{\hat{N}})\in\mathcal{U}\setminus\mathcal{U}^{f}$ and let $B:=\{\omega\in\Omega:\hat{N}(\omega)>k,\>\forall k>0\}$, then $\mathbb{P}[B]>0$. Furthermore, if $B$ holds then the switching mode $\xi$ must make an infinite number of loops and

[TABLE]

for all $k\geq 0$, by Assumptions 2.5.(iii.b) and 2.5.(i.a). However, again by Assumption 2.5.(i.a) we have444Throughout $C$ will denote a generic positive constant that may change value from line to line. $J(\emptyset)\geq-C$. Hence, $\hat{u}$ is dominated by the strategy of doing nothing and the assertion follows.∎

2.2 The Snell envelope

In this section we gather the main results concerning the Snell envelope that will be useful later on. Recall that a progressively measurable process $U$ is of class [D] if the set of random variables $\{U_{\tau}:\tau\in\mathcal{T}\}$ is uniformly integrable.

Theorem 2.4** (The Snell envelope).**

Let $U=(U_{t})_{0\leq t\leq T}$ be an $\mathbb{F}$-adapted, $\mathbb{R}$-valued, càdlàg process of class [D]. Then there exists a unique (up to indistinguishability), $\mathbb{R}$-valued càdlàg process $Z=(Z_{t})_{0\leq t\leq T}$ called the Snell envelope, such that $Z$ is the smallest supermartingale that dominates $U$. Moreover, the following holds (with $\Delta U_{t}:=U_{t}-U_{t-}$):

1. (i)

For any stopping time $\gamma$,

[TABLE] 2. (ii)

The Doob-Meyer decomposition of the supermartingale $Z$ implies the existence of a triple $(M,K^{c},K^{d})$ where $(M_{t}:0\leq t\leq T)$ is a uniformly integrable right-continuous martingale, $(K^{c}_{t}:0\leq t\leq T)$ is a non-decreasing, predictable, continuous process with $K^{c}_{0}=0$ and $(K^{d}_{t}:0\leq t\leq T)$ is non-decreasing purely discontinuous predictable with $K^{d}_{0}=0$, such that

[TABLE]

Furthermore, $\{\Delta_{t}K^{d}>0\}\subset\{\Delta_{t}U<0\}\cap\{Z_{t-}=U_{t-}\}$ for all $t\in[0,T]$. 3. (iii)

Let $\theta\in\mathcal{T}$ be given and assume that for any predictable $\gamma\in\mathcal{T}_{\theta}$ and any increasing sequence $\{\gamma_{k}\}_{k\geq 0}$ with $\gamma_{k}\in\mathcal{T}_{\theta}$ and $\lim_{k\to\infty}\gamma_{k}=\gamma$, $\mathbb{P}$-a.s, we have $\limsup_{k\to\infty}U_{\gamma_{k}}\leq U_{\gamma}$, $\mathbb{P}$-a.s. Then, the stopping time $\tau^{*}_{\theta}$ defined by $\tau^{*}_{\theta}:=\inf\{s\geq\theta:Z_{s}=U_{s}\}\wedge T$ is optimal after $\theta$, i.e.

[TABLE]

Furthermore, in this setting the Snell envelope, $Z$, is quasi-left continuous, i.e. $K^{d}\equiv 0$. 4. (iv)

Let $U^{k}$ be a sequence of càdlàg processes converging pointwisely to a càdlàg process $U$ and let $Z^{k}$ be the Snell envelope of $U^{k}$. Then the sequence $Z^{k}$ converges pointwisely to a process $Z$ and $Z$ is the Snell envelope of $U$.

In the above theorem (i)-(iii) are standard. Proofs can be found in [13] (see [22] for an English version), Appendix D in [19], [15] and in the appendix of [9]. Statement (iv) was proved in [11].

The Snell envelope will be the main tool in showing that Problem 1 has a solution.

2.3 Additional assumptions on regularity

From the definition of the Snell envelope it is clear that we need to make some further assumptions on the regularity of the involved processes. To facilitate this we define, for each $(\mathbb{t},\mathbb{b})=(t_{1},\ldots,t_{n};b_{1},\ldots,b_{n})\in\mathcal{D}^{f}$, the value process corresponding to the control $u\in\mathcal{U}$ as

[TABLE]

with $\beta_{0}:=b_{n}$.

We make the following additional assumptions:

Assumption 2.5**.**

1. (i)

For each $n\geq 0$ and each $(\eta,\mathbb{b})\in\bar{\mathcal{T}}^{n}\times\bar{\mathcal{I}}^{n}$ and $b\in\mathcal{I}^{-b_{n}}$ there is a sequence of maps $(\mathcal{U}\to\mathcal{U}:u\to\hat{u}^{l})_{l\geq 0}$ such that

[TABLE]

Furthermore, we have

[TABLE] 2. (ii)

For all $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and all $b\in\mathcal{I}^{-b_{n}}$, the process $(\mathop{\rm{ess}\,\sup}_{u\in\mathcal{U}^{k}}V^{\mathbb{t},s\vee t_{n};\mathbb{b},b,u}_{s}:0\leq s\leq T)$ is in $\mathcal{S}_{\textit{qlc}}^{2}$ for $k=0,1,\ldots$

3 A verification theorem

The method for solving Problem 1 will be based on deriving an optimal control under the assumption that a specific family of processes exists, and then showing that the family indeed does exist. We will refer to any such family of processes as a verification family.

Definition 3.1**.**

We define a verification family to be a family of càdlàg supermartingales $((Y^{\mathbb{t};\mathbb{b}}_{s})_{0\leq s\leq T}:(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f})$ such that:

1. a)

The family satisfies the recursion

[TABLE] 2. b)

The family is bounded in the sense that $\sup\limits_{u\in\mathcal{U}}\mathbb{E}[\sup\limits_{s\in[0,T]}|Y^{\tau_{1},\ldots,\tau_{N};\beta_{1},\ldots,\beta_{N}}_{s}|^{2}]<\infty$. 3. c)

For all $n\geq 1$ we have that for every $\mathbb{b}\in\bar{\mathcal{I}}^{n}$ and $\eta\in\bar{\mathcal{T}}^{n}$,

[TABLE]

and for all $b\in\mathcal{I}^{-b_{n}}$ we have

[TABLE] 4. d)

For every $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and every $b\in\mathcal{I}^{-b_{n}}$, the process $(Y^{\mathbb{t},s;\mathbb{b},b}_{s}:0\leq s\leq T)$ is in $\mathcal{S}_{\textit{qlc}}^{2}$.

The purpose of the present section is to reduce the solution of Problem 1 to showing existence of a verification family. This is done in the following verification theorem:

Theorem 3.2**.**

Assume that there exists a verification family $((Y^{\mathbb{t};\mathbb{b}}_{s})_{0\leq s\leq T}:(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f})$. Then the family is unique (i.e. there is at most one verification family, up to indistinguishability) and:

1. (i)

Satisfies $Y_{0}=\sup_{u\in\mathcal{U}}J(u)$ (where $Y:=Y^{\emptyset}$). 2. (ii)

Defines the optimal control, $u^{*}=(\tau_{1}^{*},\ldots,\tau_{N^{*}}^{*};\beta_{1}^{*},\ldots,\beta_{N^{*}}^{*})$, for Problem 1, where $(\tau_{j}^{*})_{1\leq j\leq{N^{*}}}$ is a sequence of $\mathbb{F}$-stopping times given by

[TABLE]

$(\beta_{j}^{*})_{1\leq j\leq{N^{*}}}$* is defined as a measurable selection of*

[TABLE]

and $N^{*}=\max\{j:\tau^{*}_{j}<T\}$, with $(\tau_{0}^{*},\beta^{*}_{0}):=(0,b_{0})$.

Proof. The proof is divided into three steps where we first, in steps 1 and 2, show that for any $0\leq j\leq N^{*}$ we have

[TABLE]

$\mathbb{P}$-a.s. for $s\in[\tau_{j}^{*},\tau^{*}_{j+1}]$. Then in Step 3 we show that $u^{*}$ is the optimal control estabilishing (i) and (ii). A straightforward generalization to arbitrary initial conditions $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ then gives that

[TABLE]

by which uniqueness follows.

Step 1 We start by showing that for each $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ the recursion (3.1) can be written in terms of a $\mathbb{F}$-stopping time. From (3.1) we note that, by definition, $Y^{\mathbb{t};\mathbb{b}}$ is the smallest supermartingale that dominates

[TABLE]

Now, by Assumption 2.2.(iii) and property d) in the definition of a verification family (Definition 3.1) we note that $U^{\mathbb{t};\mathbb{b}}$ is a càdlàg process of class [D] that is quasi-left continuous on $[0,T)$. Furthermore, by Assumption 2.2.(ii) and property d) we get that for any sequence $(\eta_{k})_{k\geq 0}\subset\mathcal{T}$ such that $\eta_{k}\nearrow T$, $\mathbb{P}$-a.s. we have $\lim_{k\to\infty}U^{\mathbb{t};\mathbb{b}}_{\eta_{k}}\leq U^{\mathbb{t};\mathbb{b}}_{T}$, $\mathbb{P}$-a.s. By Theorem 2.4.(iii) it thus follows that for any $\theta\in\mathcal{T}$, there is a stopping time $\gamma_{\theta}\in\mathcal{T}_{t_{n}\vee\theta}$ such that:

[TABLE]

Step 2 We now show that $Y_{0}=J(u^{*})$. We start by noting that $Y$ is the Snell envelope of

[TABLE]

where $\Psi_{0}:=\Psi(\emptyset)$, and by step 1 we thus have

[TABLE]

Moving on we pick $j\in\{1,\ldots,N^{*}\}$. For $M\geq 0$, let $z_{-1}=-1$ and $z_{k}:=kT/2^{M}$ for $k=0,\ldots,2^{M}$. Furthermore, we define the processes $(\hat{Y}^{M}_{s}:0\leq s\leq T)$ and $(\hat{U}^{M}_{t}:0\leq s\leq T)$ by

[TABLE]

and

[TABLE]

for all $s\in[0,T]$, where $\bar{\mathbb{Z}}^{j}:=\{(k_{1},\ldots,k_{j})\in\{0,\ldots,2^{M}\}^{j}:k_{1}\leq k_{2}\leq\cdots\leq k_{j}\}$. Now, for each $(k_{1},\ldots,k_{j},b_{1},\ldots,b_{j})\in\bar{\mathbb{Z}}^{j}\times\bar{\mathcal{I}}^{j}$ we have that

[TABLE]

is the product of an $\mathcal{F}_{\tau^{*}_{j}}$–measurable positive r.v. and a càdlàg supermartingale, thus, it is a càdlàg supermartingale for $s\geq\tau^{*}_{j}$. Hence, $\hat{Y}^{M}$ is the sum of a finite number of càdlàg supermartingales and thus a càdlàg supermartingale itself. By definition we find that $\hat{Y}^{M}$ dominates $\hat{U}^{M}$ which is of class [D] by Assumption 2.5.(i) and property b). To show that $\hat{Y}^{M}$ is in fact the Snell envelope of $\hat{U}^{M}$ assume that $Z$ is another càdlàg supermartingale that dominates $\hat{U}^{M}$ for all $s\in[\tau^{*}_{j},T]$. Then for each $(k_{1},\ldots,k_{j};b_{1},\ldots,b_{j})\in\bar{\mathbb{Z}}^{j}\times\bar{\mathcal{I}}^{j}$ and $s\geq\tau_{j}^{*}$, we have

[TABLE]

$\mathbb{P}$-a.s. which by (3.1) gives that

[TABLE]

Summing over all $(k_{1},\ldots,k_{j};b_{1},\ldots,b_{j})\in\bar{\mathbb{Z}}^{j}\times\bar{\mathcal{I}}^{j}$ we get $Z_{s}\geq\hat{Y}^{M}_{s}$, $\mathbb{P}$-a.s.

Noting that $\hat{Y}^{M}=Y^{\Gamma^{M}(\tau_{1}^{*},\ldots,\tau_{j}^{*});\beta_{1}^{*},\ldots,\beta_{j}^{*}}$ and using (3.2) of property c) we find that

$\sup_{s\in[0,T]}|Y_{s}^{\tau^{*}_{1},\ldots,\tau^{*}_{j};\beta^{*}_{1},\ldots,\beta^{*}_{j}}-\hat{Y}^{M}_{s}|\to 0$ in probability, as $M\to\infty$. Hence, there is a subsequence $(M_{k})_{k\geq 1}$ such that the limit taken over the subsequence is 0, $\mathbb{P}$-a.s. Furthermore, as the convergence is uniform the limit process is càdlàg.

By right-continuity of the switching costs and $\Psi$ and (3.3) of property c) we have that $\mathbb{E}[\sup_{s\in[0,T]}|U_{s}-\hat{U}^{M_{k}}_{s}|^{2}]\to 0$ as $k\to\infty$, where for notational simplicity we abuse the notation in (3.6) and let

[TABLE]

Hence, $(M_{k})_{k\geq 0}$ has a subsequence $(\tilde{M}_{k})_{k\geq 0}$ such that $\sup_{s\in[0,T]}|U_{s}-\hat{U}^{\tilde{M}_{k}}_{s}|\to 0$, $\mathbb{P}$-a.s. as $k\to\infty$. This implies that $U$ is a càdlàg process which is of class [D] by Assumption 2.5.(i) and property b).

We thus have that $\hat{U}^{\tilde{M}_{k}}$ is a sequence of càdlàg processes of class [D] that converges pointwisely to the càdlàg process $U$ of class [D] and that $\hat{Y}^{\tilde{M}_{k}}$ is the Snell envelope of $\hat{U}^{\tilde{M}_{k}}$, for all $k\geq 0$. Then by Theorem 2.4.(iv) we find that $\hat{Y}^{\tilde{M}_{k}}$ converges pointwisely to the Snell envelope Snell envelope of $U$. Hence, \Big{(}Y^{\tau^{*}_{1},\ldots,\tau^{*}_{j};\beta^{*}_{1},\ldots,\beta^{*}_{j}}_{s}:\>\tau^{*}_{j}\leq s\leq T\Big{)} is the Snell envelope of $U$.

To arrive at the second equality in (3.4) we note that the results we obtained in Step 1 implies that for any sequence $(\gamma_{l})_{l\geq 0}\subset\mathcal{T}$ with $\gamma_{l}\nearrow\gamma\in\mathcal{T}$ we have $\lim_{l\to\infty}\mathbb{E}[\hat{U}^{M}_{\gamma_{l}}]\leq\mathbb{E}[\hat{U}^{M}_{\gamma}]$ for all $M\geq 1$. Now, for all $k\geq 0$ this gives

[TABLE]

where the last term can be made arbitrarily small and we, thus, have that $\lim_{l\to\infty}\mathbb{E}[U_{\gamma_{l}}]\leq\mathbb{E}[U_{\gamma}]$ and by Theorem 2.4.(iii) we get (3.4).

By induction we get that for each $K\geq 0$,

[TABLE]

Now, arguing as in the proof of Proposition 2.3 and using property b) we find that $u^{*}\in\mathcal{U}^{f}$. Letting $K\to\infty$ and using dominated convergence we conclude that $Y_{0}=J(u^{*})$.

Step 3 It remains to show that the strategy $u^{*}$ is optimal. To do this we pick any other strategy $\hat{u}:=(\hat{\tau}_{1},\ldots,\hat{\tau}_{\hat{N}};\hat{\beta}_{1},\ldots,\hat{\beta}_{\hat{N}})\in\mathcal{U}^{f}$. By the definition of $Y_{0}$ in (3.1) we have

[TABLE]

but in the same way

[TABLE]

$\mathbb{P}$–a.s. By repeating this argument and using the dominated convergence theorem we find that $J(u^{*})\geq J(\hat{u})$ which proves that $u^{*}$ is in fact optimal. Repeating the above procedure with $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ as initial condition (3.5) follows.∎

The main difference between the above proof and the proof of Theorem 1 in the original work by Djehiche, Hamadéne and Popier [11] is that, due to the fact that the future reward at any time depends on the entire history of the control, we are forced consider a family of processes indexed by an uncountable set rather than a $q$-tuple for some finite positive $q$. Hence, we cannot simply write $Y^{\tau_{1}^{*},\ldots,\tau_{j}^{*};\beta_{1}^{*},\ldots,\beta_{j}^{*}}$ as the sum of a finite number of Snell envelopes. To arrive at the above verification theorem we therefore impose the right-continuity constraint assumed in Assumption 2.5.i. This effectively allowed us to find the two sequences of processes that approach on the one hand the value process corresponding to the optimal control and on the other hand the dominated process, in $\mathcal{S}^{2}$.

4 Existence

Theorem 3.2 presumes existence of the verification family $((Y^{\mathbb{t};\mathbb{b}}_{s})_{0\leq s\leq T}:(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f})$. To obtain a satisfactory solution to Problem 1, we thus need to establish that a verification family exists. This is the topic of the present section. We will follow the standard existence proof which goes by applying a Picard iteration (see [7, 11, 17]). We thus define a sequence $((Y^{\mathbb{t};\mathbb{b},k}_{s})_{0\leq s\leq T}:(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f})_{k\geq 0}$ of families of processes as

[TABLE]

and

[TABLE]

for $k\geq 1$.

Proposition 4.1**.**

The sequence $((Y^{\mathbb{t};\mathbb{b},k}_{s})_{0\leq s\leq T}:(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f})_{k\geq 0}$ is uniformly bounded in the sense that there is a $K>0$ such that,

[TABLE]

and for all $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and $b\in\mathcal{I}^{-b_{n}}$, we have

[TABLE]

for all $k\geq 0$.

Proof. By the definition of $Y^{\mathbb{t};\mathbb{b},k}$ we have that for any $u\in\mathcal{U}^{f}$,

[TABLE]

By Doob’s maximal inequality we have that for any $\hat{u}:=(\hat{\tau}_{1},\ldots;\hat{\beta}_{1},\ldots)\in\mathcal{U}$

[TABLE]

Taking the supremum over all $\hat{u}\in\mathcal{U}$ on both sides and using that the right hand side is uniformly bounded by Assumption 2.2.(i.a) the first bound follows.

Concerning the second claim, note that

[TABLE]

Now, arguing as above we find that

[TABLE]

where the right hand side is bounded by Assumption 2.2.(i.b). ∎

Proposition 4.2**.**

The family of processes $((Y^{\mathbb{t};\mathbb{b},k}_{s})_{0\leq s\leq T}:(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f})$ satisfies:

1. i)

For every $n\geq 1$ and every $(\eta,\mathbb{b})\in\bar{\mathcal{T}}^{n}\times\bar{\mathcal{I}}^{n}$ and $b\in\mathcal{I}^{-b_{n}}$ we have

[TABLE]

and

[TABLE]

as $l\to\infty$ uniformly in $k$. 2. ii)

For every $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and every $b\in\mathcal{I}^{-b_{n}}$, the process $(Y^{\mathbb{t},s\vee t_{n};\mathbb{b},b,k}_{s}:0\leq s\leq T)$ is in $\mathcal{S}_{\textit{qlc}}^{2}$ for $k=0,1,\ldots$

Proof. The proof will follow by induction and we use (i’) to denote the first statement without the uniformity.

For $k=0$, we have $Y^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b,0}_{\cdot}=V_{\cdot}^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b,\emptyset}\in\mathcal{S}_{\textit{qlc}}^{2}$ by Assumption 2.5.(ii) and (i’) follows from Assumption 2.5.(i). Now, assume that there is a $k^{\prime}\geq 0$ such that (i’) and (ii) holds for all $k\leq k^{\prime}$. Applying a reasoning similar to that in the proof of Theorem 3.2 we find that

[TABLE]

But then by Assumption 2.5 we find that (i’) and (ii) hold for $k^{\prime}+1$. By induction (i’) and (ii) hold for all $k\geq 0$.

It remains to show that (i) holds. By the above reasoning we find that, for each $k$ we have

[TABLE]

where the right hand side of the last inequality does not depend on $k$ and tends to zero as $l\to\infty$ by Assumption 2.5.(i). The second statement in (i) follows by an identical argument.∎

Corollary 4.3**.**

For each $k\geq 0$ and each $s\in[0,T]$ there is a $u^{k}=(\tau^{k}_{1},\ldots,\tau^{k}_{N^{k}};$ $\beta^{k}_{1},\ldots,\beta^{k}_{N^{k}})\in\mathcal{U}^{k}_{t_{n}\vee s}$, such that

[TABLE]

with $\beta^{k}_{0}=b_{0}$.

Proof. Follows from the definition of $Y^{\mathbb{t};\mathbb{b},k}$ and Propositions 4.1 and 4.2 by applying the same argument as in the proof of the verification theorem (Theorem 3.2).

Proposition 4.4**.**

For each $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$, the limit $\bar{Y}^{\mathbb{t};\mathbb{b}}:=\lim_{k\to\infty}Y^{\mathbb{t};\mathbb{b},k}$, exists as an increasing pointwise limit, $\mathbb{P}$-a.s. Furthermore, the process $\bar{Y}^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b}_{\cdot}$ is càdlàg for each $b\in\mathcal{I}^{-b_{n}}$.

Proof. Since $\mathcal{U}^{k}_{t}\subset\mathcal{U}^{k+1}_{t}$ we have that, $\mathbb{P}$-a.s.,

[TABLE]

where the right hand side is bounded $\mathbb{P}$-a.s. by Proposition 4.1. Hence, the sequence $((Y^{\mathbb{t};\mathbb{b},k}_{s})_{0\leq s\leq T}:(\mathbb{t},\mathbb{b})\in\mathcal{D})$ is increasing and $\mathbb{P}$-a.s. bounded, thus, it converges $\mathbb{P}$-a.s. for all $s\in[0,T]$.

Concerning the second claim, note that for $p\in(1,2)$, we have

[TABLE]

for all $k\geq 0$ (where the inequalities hold $\mathbb{P}$-a.s.). Now, arguing as in the proof of Proposition 4.1 we have

[TABLE]

We thus conclude that there is a $\mathbb{P}$-null set $\mathcal{N}$ such that for each $\omega\in\Omega\setminus\mathcal{N}$ we have $K(\omega)<\infty$.

By the “no-free-loop” condition (Assumption 2.2.(iiib)) and the finiteness of $\mathcal{I}$ we get that for any control $(\tau_{1},\ldots,\tau_{N};\beta_{1},\ldots,\beta_{N})$,

[TABLE]

$\mathbb{P}$-a.s. For $\omega\in\Omega\setminus\mathcal{N}$ (in the remainder of the proof $\mathcal{N}$ denotes a generic $\mathbb{P}$-null set), we thus have

[TABLE]

where $(\tau^{k}_{1},\ldots,\tau^{k}_{N^{k}};\beta^{k}_{1},\ldots,\beta^{k}_{N^{k}})\in\mathcal{U}_{s\vee t_{n}}$ is a control corresponding to $Y^{\mathbb{t},s\vee t_{n};\mathbb{b},b,k}_{s}$. This implies that for $k^{\prime}>0$ we have,

[TABLE]

Now, for all $0\leq k^{\prime}\leq k$ we have,

[TABLE]

where we introduced the process $\breve{Y}^{\mathbb{b},\mathbb{t},k,k^{\prime}}$ corresponding to the truncation $(\tau^{k}_{1},\ldots,\tau^{k}_{N^{k}\wedge k^{\prime}};\beta^{k}_{1},\ldots,\beta^{k}_{N^{k}\wedge k^{\prime}})$ of the optimal control. As the truncation only affects the performance of the controller when $N^{k}>k^{\prime}$ we have

[TABLE]

Applying Hölder’s inequality we get that for $\omega\in\Omega\setminus\mathcal{N}$,

[TABLE]

with $\frac{1}{p}+\frac{1}{q}=1$, there is thus a constant $C=C(\omega)$ such that

[TABLE]

for all $s\in[0,T]$. We conclude that for all $\omega\in\Omega\setminus\mathcal{N}$, the sequence

$(Y^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b,k}_{\cdot}(\omega))_{k\geq 0}$ is a sequence of càdlàg functions that converges uniformly which implies that the limit is a càdlàg function.∎

Proposition 4.5**.**

The family $((\bar{Y}^{\mathbb{t};\mathbb{b}}_{s})_{0\leq s\leq T}:(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f})$ is a verification family.

Proof. As $\bar{Y}^{\mathbb{t};\mathbb{b}}$ is the pointwise limit of an increasing sequence of càdlàg supermartingales it is a càdlàg supermartingale (see p. 86 in [10]). We treat each remaining property in the definition of a verification family separately:

a) Applying the convergence result to the right hand side of (4.2) and using the fact that, by Proposition 4.4,

[TABLE]

is a càdlàg process, (iv) of Theorem 2.4 gives

[TABLE]

b) Uniform boundedness was shown in Proposition 4.1.

c) We have

[TABLE]

where taking limits is interchangeable due to the uniform convergence property shown in Proposition 4.2.(i). The second statement in c), that is equation (3.3), follows by an identical argument.

d) We know from Proposition 4.4 that $\bar{Y}^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b}_{\cdot}$ is càdlàg and by Proposition 4.1 it follows that $\bar{Y}^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b}_{\cdot}\in\mathcal{S}^{2}$. It remains to show that $\bar{Y}^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b}_{\cdot}$ is quasi-left continuous. Using the notation from the proof of Proposition 4.4 we have for $k\geq 0$,

[TABLE]

for all $\omega\in\Omega\setminus\mathcal{N}$ with $\mathbb{P}(\mathcal{N})=0$. By Proposition 4.2.(ii) the first part tends to zero $\mathbb{P}$-a.s. as $j\to\infty$. Since $k$ was arbitrary and $C$ is $\mathbb{P}$-a.s. bounded the desired result follows. This finishes the proof.∎

5 Application to SDDEs with controlled volatility

We now move to the case of impulse control of SDDEs. However, we start by formalizing the hydro-power production problem proposed as a motivating example in the introduction.

5.1 Continuous time hydro-power planning

The increasing competitiveness of electricity markets calls for new operational standards in electric power production facilities. It has previously been acknowledged that optimal switching can be useful in deriving production schedules that maximize the revenue from electricity production [7, 11, 20]. Here we will extend the applicability of optimal switching by introducing a new example, the coordinated operation of hydropower plants interconnected by hydrological coupling.

We consider the situation where a central operator controls the output of two hydropower stations located in the same river (but note that the model is easily extended to consider an entire system of power stations).

We assume that Plant $i$, for $i=1,2$, has:

• •

A reservoir containing a volume $Z^{i}_{t}$ m3 of water at time $t$.

• •

A stochastic inflow $V^{i}_{t}$ m3/s to the reservoir that is modeled by a jump diffusion process.

• •

$\kappa_{i}$ turbines that can be either “in operation”, producing $p_{i}(Z^{i}_{t})$ MW by releasing $\alpha_{i}$ m3/s of water through the turbine or “idle”.

We assume that the power plants are hydrologically connected in such a way that the water that passes through Plant 1 will reach the reservoir of Plant 2 after $\delta\geq 0$ seconds.

We assume that we control the number of turbines in operation in each of the two plants. We thus let $\mathcal{I}:=\{0,1,\ldots,\kappa_{1}\}\times\{0,1,\ldots,\kappa_{2}\}$. The dynamics of the involved processes is then given by

[TABLE]

and an appropriate reward functional is

[TABLE]

where $R_{t}$ is the (stochastic) electricity price at time $t$ and $q:\mathbb{R}_{+}^{2}\to\mathbb{R}$ is the value of water (per m3) stored in the reservoirs at the end of the operation period555Note that we expect the water in Reservoir 1 to have a higher value as it can be used in both plants..

5.2 A general SDDE model

Motivated by the above example we assume that $\mathbb{F}$ is the completed filtration generated by an $d$-dimensional Brownian motion $W$ and an $d$-dimensional, independent, finite activity, Poisson random measure $\Gamma$ with intensity measure $\nu(ds;dz)=ds\times\mu(dz)$, where $\mu$ is the Lévy measure on $\mathbb{R}^{d}$ of $\Gamma$ and $\tilde{\Gamma}(ds;dz):=(\Gamma-\nu)(ds;dz)$ is called the compensated jump martingale random measure of $\Gamma$. For $u\in\mathcal{U}$, we let $X^{u,0}$ solve

[TABLE]

where $\delta>0$ is a constant and $\chi:[-\delta,0]\to\mathbb{R}^{d}$ is a deterministic càdlàg function with $\sup_{s\in[-\delta,0]}|\chi(s)|\leq C$, and define recursively

[TABLE]

Finally we let $X^{u}:=\lim_{j\to\infty}X^{u,j}$ be our controlled process666Whenever it exists, we refer to the limit process $X^{u}$ as a solution to the SDDE (5.3)-(5.5).

Remark 5.1**.**

Note that by letting $\chi_{1}\equiv b_{0}$ and taking $[h_{\beta_{j-1},\beta_{j}}]_{1}(t,x)=\beta_{j}$ and letting the first rows of $a$, $\sigma$ and $\gamma$ equal zeros we get $[X]_{1}=\xi^{u}$ which implies that the control enters all terms in the SDDE for $X^{u}$.

We consider the situation when the functional $J$ is given by

[TABLE]

We assume that the parameters of the SDDE satisfies the following conditions:

Assumption 5.2**.**

1. i)

The functions $a:[0,T]\times\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}^{d}$ and $\sigma:[0,T]\times\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}^{d}\times\mathbb{R}^{d}$ are continuous in $t$ and satisfy

[TABLE]

for all $(x,x^{\prime},y,y^{\prime})\in\mathbb{R}^{4d}$. 2. ii)

There is a $\rho(z)$, with $\int\rho^{4q}(z)\mu(dz)<\infty$ such that $\gamma:[0,T]\times\mathbb{R}^{d}\times\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}^{d}$ satisfies

[TABLE] 3. iii)

For all $(t,x)\in[0,T]\times\mathbb{R}^{d}$ and all $(b,b^{\prime})\in\bar{\mathcal{I}}^{2}$, the map $h_{b,b^{\prime}}:[0,T]\times\mathbb{R}^{d}\to\mathbb{R}^{d}$ satisfies

[TABLE]

Furthermore,

[TABLE]

for all $(x,x^{\prime})\in\mathbb{R}^{2d}$ and $(t,t^{\prime})\in[0,T]^{2}$.

Remark 5.3**.**

Note in particular that since $a$ and $\sigma$ are continuous in $t$, $a(\cdot,0,0)$ and $\sigma(\cdot,0,0)$ are uniformly bounded and Lipschitz continuity implies that

[TABLE]

We have the following result:

Proposition 5.4**.**

Under Assumption 5.2 the SDDE (5.3)-(5.5) admits a unique solution for each $u\in\mathcal{U}$. Furthermore, the solution has moments of order $4q$, i.e. \sup_{u\in\mathcal{U}}\mathbb{E}\big{[}\sup_{t\in[0,T]}|X^{u}_{t}|^{4q}\big{]}<\infty.

Proof. We first note that existence of a unique solution to the SDDE follows by repeated use of Theorem 3.2 in [1] (where existence of a unique solution to a more general controlled SDDE is shown). It remains to show that the moment estimate holds. We have $X^{u,j}=X^{u,j-1}$ on $[-\delta,\tau_{j})$ and

[TABLE]

on $[\tau_{j},T]$. By Assumption 5.2.(iii) we get, for $t\in[\tau_{j},T]$, using integration by parts, that

[TABLE]

By repeated application we find that

[TABLE]

with $\tau_{0}:=0$. Now, since $X^{u,i}$ and $X^{u,j}$ coincide on $[0,\tau_{i+1\wedge j+1})$ we have

[TABLE]

and

[TABLE]

Finally, using the Burkholder-Davis-Gundy inequality in combination with (5.6) we get

[TABLE]

where the constant $C$ does not depend on $j$ and it follows by Grönwall’s lemma that \mathbb{E}\Big{[}\sup_{t\in[0,T]}|X^{u,j}_{t}|^{4q}\Big{]} is bounded uniformly in $j$. Now, the result follows since $\tau_{j}\to T$, $\mathbb{P}$-a.s., as $j\to\infty$.∎

For each $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and each $u\in\mathcal{U}$ we let

[TABLE]

and

[TABLE]

Proposition 5.5**.**

For all $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ we have

[TABLE]

Proof. For $t\in[t_{n},T]$ we have, for $s\geq t$,

[TABLE]

Arguing as in the proof of Proposition 5.4 we find that for $s\in[\tau_{j},T]$,

[TABLE]

We thus find that, for each $u\in\mathcal{U}$,

[TABLE]

and the assertion again follows by applying Grönwall’s lemma and using Proposition 5.4.∎

To illustrate that switching does not diverge solutions we have the following useful lemma:

Lemma 5.6**.**

For $\gamma\in\mathcal{T}$ and each $u\in\mathcal{U}_{\gamma}$, let $(^{k}\!Z^{u})_{k\geq 0}$ and $X^{u}$ be processes in $\mathcal{S}^{4q}$ (with $\mathbb{E}[\sup_{s\in[0,\gamma]}|^{k}Z^{u}|^{4q}]$ uniformly bounded) that solve the SDDE (5.3)-(5.5) on $(\gamma,T]$ with control $u$ and such that

[TABLE]

as $k\to\infty$. Then,

[TABLE]

and for all $b\in\mathcal{I}^{-b_{0}}$ we have

[TABLE]

Proof. By the contraction property of $h_{.,.}$ we have that $|X_{\tau_{j}}^{u,j}-^{k}\!\!Z_{\tau_{j}}^{u,j}|<|X_{\tau_{j}}^{u,j-1}-^{k}\!\!Z_{\tau_{j}}^{u,j-1}|$. Using integration by parts we get, for $t\in[\tau_{j},T]$,

[TABLE]

Repeated application implies that

[TABLE]

Now, for $s\in(\tau_{j},T]$ we have

[TABLE]

Using Lipschitz continuity of $a,\sigma$ and $\gamma$ and the Burkholder-Davis-Gundy inequality we get

[TABLE]

where the constant $C$ does not depend on the control $u$, and by Grönwall’s inequality we have

[TABLE]

Now, applying Jensen’s inequality gives (5.8). Furthermore, we have

[TABLE]

and (5.9) follows by an identical argument.∎

We add the following assumptions on the components of the cost functional and the functions $h$.

Assumption 5.7**.**

1. (i)

The functions $f:[0,T]\times\mathbb{R}^{d}\to\mathbb{R}$ and $g:\mathbb{R}^{d}\to\mathbb{R}$ are both locally Lipschitz in $x$. Furthermore, there are constants $q>1$ and $K>0$ such that

[TABLE]

for all $(t,x)\in[0,T]\times\mathbb{R}^{d}$. 2. (ii)

For all $b\in\mathcal{I}$ we have

[TABLE]

*for all $x\in\mathbb{R}^{d}$. * 3. (iii)

There is a constant $\kappa>0$ such that for any sequence $(b_{1},\ldots,b_{j})\in\bar{\mathcal{I}}^{j}$ with $j>\kappa$ there is a subsequence $1=\iota_{1}<\cdots<\iota_{j^{\prime}}=j$ with $j^{\prime}\leq\kappa$ and $(b_{\iota_{1}},\ldots,b_{\iota_{j^{\prime}}})\in\bar{\mathcal{I}}^{j^{\prime}}$ for which

[TABLE]

It is straightforward to see that with the above assumptions the $\Psi$ defined by

[TABLE]

satisfies Assumption 2.2. The remainder of this section is devoted to showing that $\Psi$ also satisfies Assumption 2.5, guaranteeing the existence of an optimal control to the problem of maximizing $J$.

Proposition 5.8**.**

For each $n\geq 1$ and each $(\eta,\mathbb{b})\in\bar{\mathcal{T}}^{n}\times\bar{\mathcal{I}}^{n}$ and $b\in\mathcal{I}^{-b_{n}}$ there is a map $(\mathcal{U}\to\mathcal{U}:u\to\hat{u}^{l})_{l\geq 1}$ such that

[TABLE]

and

[TABLE]

Furthermore, we have

[TABLE]

and

[TABLE]

Proof. To simplify notation we let $(\zeta_{i})_{1\leq i\leq n}$ denote $\Gamma^{l}(\eta)$ and let $X$ and $Z$ (resp. $X^{j}$ and $Z^{j}$) denote $X^{\eta;\mathbb{b},u}_{t}$ resp. $X^{\Gamma^{l}(\eta);\mathbb{b},\hat{u}^{l}}$ (resp. $X^{\eta;\mathbb{b},u,j}$ and $X^{\Gamma^{l}(\eta);\mathbb{b},\hat{u}^{l},j}$). Furthermore, we let $U^{*}_{t}:=\sup_{s\in[0,t]}|U_{s}|$ be the running maximum of the process $|U|$.

We have:

i) $X_{t}=Z_{t}$, for all $t\in[0,\eta_{1})$, $\mathbb{P}$-a.s.

ii) On $[\eta_{1},\zeta_{1})$ we have $|X_{t}-Z_{t}|\leq(X)_{T}^{*}+(Z)_{T}^{*}$.

iii) If $\eta_{j}\leq\zeta_{1}$, then $\zeta_{j}=\zeta_{j-1}=\cdots=\zeta_{1}$.

Letting $M_{1}:=\max\{j\geq 1:\eta_{j}\leq\zeta_{1}\}$ we get

[TABLE]

Hence,

[TABLE]

But $X^{0}_{\zeta_{1}}=Z^{0}_{\zeta_{1}}$ and by induction it follows that

[TABLE]

If we iteratively define $M_{i}:=\max\{j>M_{i-1}:\eta_{j}\leq\zeta_{M_{i-1}+1}\}$, for $i=1,\ldots n_{M}$ with $M_{n_{M}}=n$ and $M_{0}:=0$. Then we get, in the same manner,

[TABLE]

Now on $[{\zeta_{M_{i}}},T]$ we have

[TABLE]

Put together we find that for $t\in[\zeta_{M_{i}},T]$ we have

[TABLE]

Applying Thm 66, p. 339 in [26] and Lipschitz continuity iteratively gives

[TABLE]

By Grönwall’s inequality and point ii) above we find that

[TABLE]

Moving on we consider the possibility of interventions in the period $[\eta_{n},\zeta_{n})$. Let $N^{\prime}:=\max\{j\geq 0:\tau_{j}<\zeta_{n}\}$ and note that if $N^{\prime}>\kappa$, then there is a subsequence $(\iota_{j})_{j=1}^{\kappa^{\prime}}$ with $1\leq\iota_{1}<\cdots<\iota_{\kappa^{\prime}}=N^{\prime}$ with $\kappa^{\prime}\leq\kappa$ and $(b_{n},\beta_{\iota_{1}},\ldots,\beta_{\iota_{\kappa^{\prime}}})\in\bar{\mathcal{I}}^{\kappa^{\prime}+1}$ such that, for all $(t,x)\in[0,T]\times\mathbb{R}^{d}$,

[TABLE]

We then let777For $k\geq 1$ we denote by $\mathbb{1}_{k}$ the vector of $k$ ones. $\hat{u}^{l}=(\hat{\tau}_{1},\ldots,\hat{\tau}_{\hat{N}};\hat{\beta}_{1},\ldots,\hat{\beta}_{\hat{N}}):=$

$(\zeta_{n}{\mathbb{1}}_{\kappa^{\prime}},\tau_{N^{\prime}+1},\ldots,\tau_{N};\beta_{\iota_{1}},\ldots,\beta_{\iota_{\kappa^{\prime}}},\beta_{N^{\prime}+1},\ldots,\beta_{N})$. Arguing as above, we find that

[TABLE]

We now turn to the total revenue and let

[TABLE]

By right continuity of the switching costs, we find that

[TABLE]

$\mathbb{P}$-a.s. The difference in revenue can then be written

[TABLE]

By local Lipschitz continuity of $f$ and $g$ we get that, for each $K>0$ there is a $C>0$ such that $|f(t,x)-f(t,x^{\prime})|\leq C|x-x^{\prime}|$ and $|g(x)-g(x^{\prime})|\leq C|x-x^{\prime}|$ on $|x|+|x^{\prime}|\leq K$. This gives us the relation

[TABLE]

where $A:=\{\omega\in\Omega:\int_{0}^{T}C|X_{s}-Z_{s}|^{2}ds+C|X_{T}-Z_{T}|^{2}>-\Lambda\}$. Doob’s maximal inequality then gives that

[TABLE]

where we have used Hölder’s inequality and the moment estimate in Proposition 5.4 to arrive at the last inequality. For any $M>0$ we thus have

[TABLE]

Concerning the first term, we have that $\mathbbm{1}_{[N^{\prime}\leq M]}|X_{s}-Z_{s}|\leq|\tilde{X}_{s}-\tilde{Z}_{s}|$, where $\tilde{X}=X$ and $\tilde{Z}=Z$ on $[N^{\prime}\leq M]$. On $[N^{\prime}>M]$ we let $\tilde{X}:=X^{\eta;\mathbb{b},\tilde{u}}$ with

[TABLE]

and $\tilde{Z}:=X^{\eta;\mathbb{b},\tilde{u}^{l}}$ where $\tilde{u}^{l}$ is obtained from $\tilde{u}$ as $\hat{u}^{l}$ was obtained from $u$. Now, we proceed as above and get for each $M\geq\kappa$, that

[TABLE]

By (5.14) and (5.8) of Lemma 5.6 we then find that for each $M\geq\kappa$, the first term on the right hand side in (5.17) goes to 0 as $l\to\infty$. Concerning the second term we have, again by Hölder’s inequality and Proposition 5.4, that

[TABLE]

Now, $A\subset\{\omega:C(X_{T}^{*}+Z^{*}_{T})>-\Lambda\}$, where $C>0$ does not depend on $l$. For $l$ sufficiently large we thus see, by (5.16) and Chebyshev’s inequality, that the probability on the right hand side can be made arbitrarily small by choosing $M$ sufficiently large. For the third term we note that

[TABLE]

where the right hand side goes to 0 as $l\to\infty$ by right-continuity of the switching costs. Finally, the last term of (5.17) can be made arbitrarily small by choosing $K$ large.

Concerning the second claim we note that with $X=X^{\eta,s\vee\eta_{n},\mathbb{b},b,u}$ and $Z=X^{\Gamma^{l}(\eta),s\vee\Gamma^{l}(\eta_{n}),\mathbb{b},b,u}$ the relation in (5.15) is replaced by

[TABLE]

Hence, appealing to (5.9) of Lemma 5.6, right-continuity and the result in Proposition 5.5 the first second and last terms in the equivalent to (5.17) tends to 0 as $l\to\infty$ and (5.12) follows.

The last two statements given in equations (5.12)-(5.13) follow by a similar reasoning while noting that in this case $N^{\prime}=0$ which implies that $\Lambda=0$, $\mathbb{P}$-a.s.∎

Lemma 5.9**.**

For all $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and $k\geq 0$ we have

[TABLE]

$\mathbb{P}$-a.s. as $t^{\prime}\searrow t$.

Proof. Starting with $k=0$ we note that for $t^{\prime}\geq t$ we have

[TABLE]

which gives

[TABLE]

For $k>0$ and $u\in\mathcal{U}^{k}_{t}$ we have, for $i\leq k$

[TABLE]

and

[TABLE]

which gives

[TABLE]

Repeated application renders

[TABLE]

Furthermore, we have

[TABLE]

where the right hand side tends to zero $\mathbb{P}$-a.s. as $t^{\prime}\searrow t$ by $\mathbb{P}$-a.s. boundedness of

$\sup_{u\in\mathcal{U}}\sup_{r\in[t_{n},T]}|(X^{\mathbb{t},r;\mathbb{b},b,u})^{*}_{T}|^{4}$. Arguing as in the proof of Lemma 5.6 we find that

[TABLE]

and the assertion follows by right continuity of $X$.∎

Lemma 5.10**.**

For all $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and all $b\in\mathcal{I}^{-b_{n}}$ we have whenever $\gamma_{j}\nearrow\gamma\in\mathcal{T}_{t_{n}}$, with $(\gamma_{j})_{j\geq 0}\subset\mathcal{T}_{t_{n}}$, that

[TABLE]

for all $0\leq k<\infty$.

Proof. Arguing as in the proof of the previous lemma we find that

[TABLE]

Furthermore, by Hölder’s inequality we have

[TABLE]

where $\frac{1}{p}+\frac{1}{q}=1$. Now, by definition $\gamma$ is a predictable stopping time and the jump part of our SDDE is $\mathbb{P}$-a.s. constant at predictable stopping times. We can, thus, apply Lemma 5.6 and the assertion follows.∎

Proposition 5.11**.**

*For all $(\mathbb{t},\mathbb{b})\in\mathcal{D}^{f}$ and all $b\in\mathcal{I}^{-b_{n}}$, the process

$(\mathop{\rm{ess}\,\sup}_{u\in\mathcal{U}^{k}}V^{\mathbb{t},s\vee t_{n};\mathbb{b},b,u}_{s}:0\leq s\leq T)$ is in $\mathcal{S}_{\textit{qlc}}^{2}$ for all $k\geq 0$.*

Proof. Let $Y^{\mathbb{t};\mathbb{b},k}_{t}:=\mathop{\rm{ess}\,\sup}_{u\in\mathcal{U}^{k}}V^{\mathbb{t};\mathbb{b},u}_{t}$. To show that $Y^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b,k}_{\cdot}$ has a càdlàg version we consider

[TABLE]

where the second term on the right hand side goes to zero $\mathbb{P}$-a.s. as $t^{\prime}\searrow t$ by uniform integrability and right continuity of the filtration. Concerning the first term we have

[TABLE]

for each $K>0$, by the local Lipschitz property of $f$ and $g$. Concerning the last term Doob’s maximal inequality gives, for fixed $u\in\mathcal{U}^{k}$,

[TABLE]

Applying Hölder’s inequality to the right hand side and taking the supremum over $\mathcal{U}$, we get

[TABLE]

Now, by Chebyshev’s inequality and Proposition 5.5,

$\sup_{u\in\mathcal{U}}\mathbb{P}[\sup_{r\in[t_{n},T]}(X^{\mathbb{t},r;\mathbb{b},b,u})_{T}^{*}\geq K]$ can be made arbitrarily small by choosing $K$ large. By monotonicity, it follows that the last term in (5.18) tends to zero, $\mathbb{P}$-a.s. as $K\to\infty$. We conclude that $Y^{\mathbb{t},t^{\prime};\mathbb{b},b,k}_{t^{\prime}}$ tends to $Y^{\mathbb{t},t;\mathbb{b},b,k}_{t}$, $\mathbb{P}$-a.s. when $t^{\prime}\searrow t$ by right continuity of the switching costs in combination with Lemma 5.9 and it follows that $Y^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b,k}_{\cdot}$ has a càdlàg version.

Arguing as above we have that

[TABLE]

Letting $j\to\infty$ the last term tends to zero $\mathbb{P}$-a.s. by uniform integrability and quasi-left continuity of the filtration. Concerning the first term we have (where we for notational convenience assume that $\gamma,\gamma_{j}\in\mathcal{T}_{t_{n}}$)

[TABLE]

where the right hand side can be made arbitrarily small by Lemma 5.10 and quasi-left continuity of the switching costs. We conclude that

[TABLE]

which implies that $Y^{\mathbb{t},\gamma_{j}\vee t_{n};\mathbb{b},b,k}_{\gamma_{j}}\to Y^{\mathbb{t},\gamma\vee t_{n};\mathbb{b},b,k}_{\gamma}$ in probability. Now since $Y^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b,k}_{\cdot}$ has left limits it follows that $Y^{\mathbb{t},\gamma_{j}\vee t_{n};\mathbb{b},b,k}_{\gamma_{j}}\to Y^{\mathbb{t},\gamma\vee t_{n};\mathbb{b},b,k}_{\gamma}$, $\mathbb{P}$-a.s. and we conclude that $Y^{\mathbb{t},\cdot\vee t_{n};\mathbb{b},b,k}_{\cdot}\in\mathcal{S}_{\textit{qlc}}^{2}$.∎

By the above results we conclude that an optimal control for the hydropower planning problem does exist (under the assumptions detailed in this section). With a few notable exceptions (see *e.g. * [3, 4] in the case of singular control problems and Chapter 7 in [25] for examples of solvable impulse control problems) finding explicit solutions to impulse control problems is difficult. Instead we often have to resort to numerical methods to approximate the optimal control. A plausible direction for obtaining numerical approximations of solutions to the hydropower operators problem would be to further develop the Monte Carlo technique originally proposed for optimal switching problems in [7] (and later analyzed in [2]) to obtain polynomial approximations of $Y^{\mathbb{t},\mathbb{b}}$. Another possibility would be to apply the Markov-Chain approximations for stochastic control problems of delay systems developed in [21]. However, a thorough investigation of either direction is out of the scope of the present work and will be left as a topic of future research.