Verification theorems for stochastic optimal control problems in Hilbert spaces by means of a generalized Dynkin formula

TL;DR

This paper introduces a new approach to verification theorems in infinite-dimensional stochastic control, utilizing a generalized Dynkin formula that relaxes the need for strong solution assumptions.

Contribution

The paper presents a novel method for verification theorems in stochastic control that does not require the mild solution to be a strong solution, using a new Dynkin formula.

Findings

Developed a new Dynkin formula for infinite-dimensional stochastic processes.

Proved verification theorems without requiring strong solutions.

Applicable to Ornstein-Uhlenbeck processes with mild solutions.

Abstract

Verification theorems are key results to successfully employ the dynamic programming approach to optimal control problems. In this paper we introduce a new method to prove verification theorems for infinite dimensional stochastic optimal control problems. The method applies in the case of additively controlled Ornstein-Uhlenbeck processes, when the associated Hamilton-Jacobi-Bellman (HJB) equation admits a mild solution. The main methodological novelty of our result relies on the fact that it is not needed to prove, as in previous literature, that the mild solution is a strong solution, i.e. a suitable limit of classical solutions of the HJB equation. To achieve our goal we prove a new type of Dynkin formula, which is the key tool for the proof of our main result.

Equations

$$dX(t)=\big{[}AX(t)+GL(u(t))\big{]}\,dt+\sigma\,dW(t),$$

$${\mathbb{E}}\left[\int_{0}^{\infty}e^{-\lambda s}l\big{(}X(s),u(s)\big{)}\,ds\right],$$

$$\lambda v(x)-\frac{1}{2}\text{mbox}\mathrm{Tr}[\sigma {\sigma }^{\ast }{D}^{2}v(x)]-{⟨Ax,Dv(x)⟩}_H-F_0(x,D^{G}v(x))=0,$$

$$F_0(x,D^{G}v(x))=\underset{u\in \Lambda }{\inf }\left\{{⟨L(u),D^{G}v(x)⟩}_K+l(x,u)\right\},$$

$$|\phi {|}_{\infty }=\underset{x\in U}{\sup }|\phi (x){|}_V.$$

$$|T|_{{\mathcal{L}}_{1}(H)}\coloneqq\sum_{k=1}^{\infty}\big{\langle}(T^{*}T)^{1/2}e_{k},e_{k}\big{\rangle}_{H}$$

$$\big{|}T\big{|}_{{\mathcal{L}}_{2}(H)}\coloneqq\left(\sum_{k=0}^{\infty}\big{|}Te_{k}\big{|}_{K}^{2}\right)^{1/2}$$

$$\big{\langle}T,S\big{\rangle}_{{\mathcal{L}}_{2}(H,K)}\coloneqq\sum_{k=0}^{\infty}\big{\langle}Te_{k},Se_{k}\big{\rangle}_{K},$$

$$\big{|}X\big{|}_{\mathcal{M}_{\mathcal{P}}^{p,T}(U)}:=\left(\int_{0}^{T}\mathbb{E}\left[|X(s)|_{U}^{p}\right]ds\right)^{1/p}<\infty.$$

$$[0,T]\to L^p(\Omega ,U),t\mapsto X(t)$$

$$\big{|}X\big{|}_{\mathcal{K}_{\mathcal{P}}^{p,T}(U)}:=\sup_{s\in[0,T]}\left(\mathbb{E}|X(s)|_{U}^{p}\right)^{1/p}.$$

$$\lim_{|h|_{H}\rightarrow 0}\frac{\big{|}f\left(x+h\right)-f\left(x\right)-\langle Df(x),h\rangle_{H}\big{|}}{|h|_{H}}=0.$$

$$\lim_{k\in{\mathcal{D}}(G),\,|k|_{K}\rightarrow 0}\frac{\big{|}f\left(x+Gk\right)-f\left(x\right)-\langle D^{G}f(x),k\rangle_{K}\big{|}}{|k|_{K}}=0.$$

$$\lim_{t\rightarrow 0}\frac{f(x+tGk)-f(x)}{t}=\big{\langle}D^{G}f(x),k\big{\rangle}_{K},\ \ \ \ \forall k\in{\mathcal{D}}(G);$$

$$\underset{t\to 0}{\lim }\frac{f(x+tGk)-f(x)}{t}=⟨k^{\prime },k{⟩}_K,\text{mbox}uniformlyink\in \mathcal{D}(G)\cap B_K(0,R),\forall R>0,$$

$$D^{G}f\left(x\right)=G^{\ast }Df\left(x\right).$$

$$\lim_{s\rightarrow 0}\frac{f\left(x+sGk\right)-f\left(x\right)}{s}=\big{\langle}Df(x),Gk\big{\rangle}_{H},\quad\forall k\in{\mathcal{D}}(G).$$

$$\lim_{s\rightarrow 0}\frac{f\left(x+sGk\right)-f\left(x\right)}{s}=\big{\langle}D^{G}f(x),k\big{\rangle}_{K},\quad\forall k\in{\mathcal{D}}(G).$$

$$\big{|}\left\langle Df\left(x\right),Gk\right\rangle_{H}\big{|}=\big{|}\big{\langle}D^{G}f(x),k\big{\rangle}_{K}\big{|}\leq\big{|}D^{G}f(x)\,\big{|}_{K}|k|_{K},\quad\forall k\in{\mathcal{D}}(G).$$

$$\begin{cases}dX(t)=\big{[}AX(t)+GL(u(t))\big{]}\,dt+\sigma\,dW(t),\ \ \ t\geq 0,\\ X(0)=x,\end{cases}$$

$${\int }_0^t{s}^{-2\gamma }\mathrm{Tr}\left[e^{sA}\sigma {\sigma }^{\ast }{e}^{s{A}^{\ast }}\right]ds<\infty \forall t\ge 0.$$

$${\left|\overline{e^{sA}G}\right|}_{\mathcal{L}(K,H)}\le C_G(s^{-\beta }\vee 1)e^{a_{G}s}\forall s>0.$$

$$\overline{e^{(s+t)A}G}=e^{sA}\overline{e^{tA}G},\forall t>0,\forall s\ge 0.$$

$$p\in \left(\frac{1}{1-\beta },+\infty \right),$$

$${\mathcal{U}}_p:=\left\{u:\Omega \times [0,+\infty )\to \Lambda \text{mbox}prog.meas.ands.t.{\int }_0^t\mathbb{E}\left[|u(s){|}_U^p\right]ds<\infty \forall t\ge 0\right\}.$$

$$F(t):={\int }_0^{t}g(t-s)f(s)ds,t\in {\mathbb{R}}^{+},$$

$$[0,t)\to V,s\mapsto g(t-s)f(s),$$

$$h_1:(0,t]\times E\to V,h_1(s,e)=g(s)e;h_2:[0,t)\to (0,t]\times E,h_2(s)=(t-s,f(s)).$$

$${\int }_0^t|g(t-s)f(s){|}_{V}ds\le {\int }_0^t(t-s{)}^{-\beta }|f(s){|}_{V}ds\le {\left({\int }_0^t(t-s{)}^{-\beta \frac{p}{p-1}}ds\right)}^{\frac{p-1}{p}}|f{|}_{L^p([0,T];\mathbb{R})}={\left(\frac{t^{\kappa }}{\kappa }\right)}^{\frac{p-1}{p}}|f{|}_{L^p([0,T];\mathbb{R})}.$$

$$F_{\epsilon }(t):={\int }_0^{t-\epsilon }g(t-s)f(s)ds,t\in [t_0,T].$$

Full text

Key words: Stochastic optimal control, infinite dimensional HJB equations, Dynkin’s formula, transition semigroups, verification theorems, optimal feedbacks.

AMS classification: 93E20 (Optimal stochastic control); 70H20 (Hamilton-Jacobi equations); 65H15 (Stochastic partial differential equations); 49L20 (Dynamic programming method); 49N35 (Optimal feedback synthesis).

Acknowledgements. The authors are sincerely grateful to Franco Flandoli, Ben Goldys and Mauro Rosestolato for fruitful discussions on Subsection 6.1.4 and Remark 6.2. The authors are also grateful to an anonymous referee for careful scrutiny and useful suggestions that led to an improved version of the paper.

1. 1 Introduction

2. 2 Preliminaries

3. 2.1 Spaces and notation

4. 2.2 $G$-derivative

5. 3 Formulation of the stochastic optimal control problem

6. 4 Generalized Dynkin’s formula

7. 4.1 Transition semigroups, generators and $G$-derivatives

8. 4.2 Proof of the generalized Dynkin’s formula

9. 5 HJB equation, verification theorem and optimal feedbacks

10. 5.1 Verification theorem

11. 5.2 Optimal feedback controls

12. 6 Applications

13. 6.1 Neumann Boundary control of a stochastic heat equation with additive noise

14. 6.1.1 Problem setup

15. 6.1.2 Infinite dimensional setting

16. 6.1.3 HJB equation and verification theorem

17. 6.1.4 Optimal Feedback Controls

18. 6.2 Stochastic optimal control with delay in the control variable

19. A Appendix

1 Introduction

In this paper we introduce a new technique, based on a generalized Dynkin formula, to prove verification theorems for stochastic optimal control problems over infinite horizon in Hilbert spaces.

Verification theorems are key results to enable to solve in a closed way optimal control problems through the dynamic programming approach. Once a solution (in some sense to be precised) of the associated HJB equation is known to exists, the verification theorem provides a sufficient (sometimes also necessary) condition of optimality, which can be used to find optimal controls in feedback forms through the so called closed loop equation. In the stochastic case, when the solution $v$ is sufficiently smooth, the proof of such theorem is substantially based on an applying the Dynkin formula to the function $v$ and to the state process. In our framework of discounted time-homogeneous infinite horizon problems the dependence on time is known, so the HJB equation is elliptic and $v$ only depends on the state variable. Hence, in the finite dimensional case, to employ the classical Dynkin formula, it is needed to know that $v\in C^{2}$. Fortunately, in the finite dimensional case, due to the presence of a powerful regularity theory (at least for nondegenerate second order HJB equations) there is a wide class of problems for which actually $v$ is known to enjoy this regularity, hence the classical Dynkin formula applies and the verification theorem can be proved. On the other hand, if $v$ is not known to be sufficiently smooth (i.e. when $v$ is known to be only a viscosity solution), still in the finite dimensional case, other techniques have been developed to overcome the fact that the classical Dynkin formula is not applicable. We mention the following techniques.

The technique developed in [33], dealing with viscosity solutions. In this case, the classical Dynkin formula is applied to test functions and only some weak results are obtained.

• -

The technique developed in [41]. Here a solution $v\in C^{1}$ is obtained through the solution of a suitable backward SDE (BSDE). This technique applies to semilinear HJB equations and provides the verification theorem as a byproduct of the construction itself of the solution $v$. The latter feature is particularly meaningful, as it allows to completely bypass the problem of second order regularity of $v$ and the application of the classical Dynkin formula. On the other hand, the powerfulness of this approach is partly limited by the fact that it can be applied only when a structural condition is verified by the control operator.

• -

The technique developed in [32]: here $v$ is studied and treated as a strong solution, i.e. as a suitable limit of classical solutions.

When the state space $H$ is infinite dimensional the situation is much worse. First of all, the regularity needed to apply the classical Dynkin formula (see, e.g., [10, Sec. 4.4]) is very demanding and does not allow to deal with many applied examples proposed and only partly studied in the literature. This is partly due to additional regularity assumptions on the coefficients needed in infinite dimension, partly due to the lack of a satisfactory regularity theory in infinite dimension. Hence, elaborating alternative methods is considerably more important than in the finite dimensional case. Clearly, the first attempt consists in trying to extend the techniques developed in the finite dimensional case to infinite dimensional one. On this side, so far the state of the art can be basically depicted as follows.

• (a)

There are no results concerning the case when $v$ is a viscosity solution.

• (b)

Results with the BSDE approach have been elaborated in various papers, see e.g. [21] in the infinite horizon case, but always under the structural condition. The latter requirement leaves out the treatments of important cases like boundary control of stochastic PDEs or delayed control of SDEs.

• (c)

Results dealing with strong solutions are available in [31] and in [5].

The results we provide here are closer, in the conclusions, to the results mentioned in item (c) above. With respect to them, ours have a larger range of applicability and, not only in this sense, can be seen as a significant improvement of this technique, as we will comment more precisely afterwards.

We stress the fact that our method to prove the verification theorem is a novelty also in finite dimension: our results may be useful to treat also finite dimensional problems where only partial regularity properties of the value function are known. Here we focus on the infinite dimensional case where the application is more meaningful.

We now illustrate the results and the novelties of our paper. We consider a class of stochastic optimal control problems in a real separable Hilbert space $H$, where the noise is additive and the control only appears in an additive form in the drift term. More precisely, the state equation is

[TABLE]

where $A:{\cal{D}}(A)\subseteq H\rightarrow H$, $G:K\rightarrow H$, $L:\Lambda\rightarrow K$, $\sigma:\Xi\rightarrow H$ are suitable operators, with $K,\Xi$ being other real separable Hilbert spaces and $\Lambda$ being a Polish space; $W$ is a $\Xi$-valued cylindrical Browian motion; $u$ is the control process taking values in $\Lambda$; $X$ is the state process taking values in the Hilbert space $H$. The stochastic control problem consists in minimizing, over a set of admissible control processes, a cost functional in the form

[TABLE]

where $\lambda>0$ is a discount factor and $l$ is a suitable real valued function. In this case the associated HJB equation is an elliptic semilinear PDE in the space $H$:

[TABLE]

where

[TABLE]

where $D^{G}v$ denotes the $G$-gradient of a function $v:H\rightarrow\mathbb{R}$ (see Subsection 2.2). Under reasonable assumptions, it is proved in [16] that such HJB equation admits a unique mild solution, i.e. a solution of a suitable integral form of the above equation. Such solution admits $G$-gradient, i.e. verifies the minimal differentiability requirement to give sense to the nonlinear Hamiltonian term $F_{0}$ in HJB above. Once one proves the existence of a mild solution $v$ to the associated HJB equation, the approach of item (c) would require three nontrivial technical steps: first, proving that such a mild solution is indeed a strong solution (limit, in a suitable sense, of classical solution); second, applying Dynkin formula to the approximating classical solutions; third, passing to the limit the Dynkin formula. As one may expect, passing through all these steps requires additional hypotheses that may be nontrivial to check in practice (see e.g. [31]). Our goal here is to bypass these steps through an alternative path. In fact, we show that the role of strong solutions is not essential. Indeed, relying on the theory of $\pi$-semigroups (see e.g. [14, Appendix B] and [43]), we prove a generalized (abstract) Dynkin formula — deserving interest in itself — which can be directly applied to mild solutions. The proof is quite involved and this is the reason why we consider here the case of stochastic control of equation of type (1.1), where the uncontrolled part of the state equation is of Ornstein-Uhlenbeck type333It is worth to stress that, even if in the case of Ornstein-Uhlenbeck dynamics the approach of strong solutions has already been succesfully applied (see [31]), the method used here, other than being original, seems to be extendable to more general structures of state equations, where the strong solution approach would fail.. Then, relying on this formula, we straightly prove a verification theorem. The new results on $G$-derivatives provided in [16] (see also [14, Ch. 4]) enable us to apply our method to more general examples than the ones treated by the current literature; in particular, to cases where the structural condition required at item (b) above is not verified (see Section 6).

The main results of the paper are the abstract Dynkin formula (Theorem 4.8); the verification theorem (Theorem 5.6); the consequent Corollary 5.7 on sufficient conditions for the existence of optimal control processes in feedback form. Moreover, since the existence of optimal feedback controls might be is easier to obtain when the optimal control problem is considered in the weak formulation, i.e., letting also the stochastic basis to vary, we also provide Corollary 5.8 in this direction. We underline that we do not provide general results on the existence of optimal control processes in feedback form, as such results strongly depend on the specific case at hand. To this regard, in Section 6 — where we deal with two specific applications: optimal boundary control (of Neumann type) of the stochastic heat equation and optimal control of SDEs with delay in the control variable — we provide for the first example some results and comments on the existence of optimal feedback control processes.

The paper is organized as follows. After some preliminaries in Section 2 on spaces, notation and the notion of $G$-derivative recently extended in [16], we introduce our family of control problems in Section 3. Section 4 is devoted to prove our new Dynkin formula (Theorem 4.8), the methodological core of the paper. In Section 5 we prove our main results on the control problem: in Subsection 5.1, the verification theorem (Theorem 5.6); in Subsection 5.2, Corollary 5.7 on optimal feedbacks. Section 6 is devoted to illustrate the applications of our results to the aforementioned examples. Finally the Appendix is devoted to prove few technical results needed to prove our Dynkin formula.

2 Preliminaries

In this section we provide some preliminaries about spaces and notation used in the rest of the paper and recall from [16] the notion of $G$-derivative. We restrict the treatment of $G$-derivative to the case of real valued functions defined on Hilbert spaces and to constant operator maps $G$. This will be enough for the purposes of the present paper. For a more general theory and more details we refer to the aforementioned paper [16].

2.1 Spaces and notation

Measurable bounded and continuous functions.

All the topological spaces are intended endowed with their Borel $\sigma$-algebra, denoted by ${\mathcal{B}}$. By measurable set (function), we always intend a Borel measurable set (function). If $U$ is a topological space and $V$ is a topological vector space, we denote by $B_{b}(U,V)$ the set of bounded measurable functions from $U$ to $V$ and by $C_{b}(U,V)$ the set of bounded continuous functions from $U$ to $V$. If $V=\mathbb{R}$, we drop it in the latter notation. If $V$ is complete, the spaces $B_{b}(U,V)$ and $C_{b}({U},V)$ are Banach spaces when endowed with the norm

[TABLE]

Hilbert spaces.

Let $H$ be a Hilbert space. We denote its norm by $|\cdot|_{H}$ and its inner product by by $\left\langle\cdot,\cdot\right\rangle_{H}$. We omit the subscript if the context is clear and if $H=\mathbb{R}$. If a sequence $(x_{n})_{n\in\mathbb{N}}\subseteq H$, converges to $x\in U$ in the norm (strong) topology we write $x_{n}\rightarrow x$.

We denote by $H^{*}$ the topological dual of $H$, i.e. the space of all continuous linear functionals defined on $H$. We always identify $H^{*}$ with $H$ through the standard Riesz identification.

Linear operators.

Let $H,K$ be real separable Hilbert spaces. We denote by $\mathcal{L}(H,K)$ the set of all bounded (continuous) linear operators $T:H\rightarrow K$ with norm $|T|_{\mathcal{L}(H,K)}:=\sup_{x\in H,x\neq 0}\frac{|Tx|_{K}}{|x|_{H}}$, using for simplicity the notation $\mathcal{L}(H)$ when $H=K$. Moreover, we denote by ${\mathcal{L}}_{u}(H,K)$ the space of closed densely defined and possibly unbounded linear operators $T:\mathcal{D}(T)\subseteq H\rightarrow K$, where ${\mathcal{D}}(T)$ denotes the domain. We recall that ${\mathcal{D}}(T)$ is a Hilbert space when endowed with the graph norm $|x|_{{\mathcal{D}}(T)}=|x|_{H}+|Tx|_{K}$. The range of an operator $T\in{\mathcal{L}}_{u}(H,K)$ is denoted by ${\mathcal{R}}(T)$. Clearly, ${\mathcal{L}}(H,K)\subseteq{\mathcal{L}}_{u}(H,K)$. Given $T\in{\mathcal{L}}_{u}(H,K)$, we denote its adjoint operator by $T^{*}:\mathcal{D}(T^{*})\subseteq K\rightarrow H$.

We denote by ${\mathcal{L}}_{1}(H)$ the set of trace class operators, i.e. the operators $T\in{\mathcal{L}}(H)$ such that, given an orthonormal basis $\{e_{k}\}_{k\in\mathbb{N}}$ of $H$, the quantity

[TABLE]

is finite (see [45, Sec. VI.6]). The latter quantity is independent of the basis chosen and defines a norm making ${\mathcal{L}}_{1}(H)$ a separable Banach space. The trace of an operator $T\in{\mathcal{L}}_{1}(H)$ is denoted by $\mbox{Tr}[T]$, i.e. $\mbox{Tr}[T]\coloneqq\sum_{k=0}^{\infty}\langle Te_{k},e_{k}\rangle_{U}$. The latter quantity is finite and, again, independent of the basis chosen. We denote by ${\mathcal{L}}_{1}^{+}(U)$ the subset of ${\mathcal{L}}_{1}(H)$ of self-adjoint nonnegative (trace class) operators on $H$. Note that, if $T\in{\mathcal{L}}_{1}^{+}(H)$, then $\mbox{Tr}[T]=|T|_{{\mathcal{L}}_{1}(U)}$.

We denote by ${\mathcal{L}}_{2}(H,K)$ (subset of ${\mathcal{L}}(H,K)$) the space of Hilbert-Schmidt operators from $H$ to $K$, i.e the spaces of operators such that, given an orthonormal basis $\{e_{k}\}_{k\in\mathbb{N}}$ of $H$, the quantity

[TABLE]

is finite (see [45, Sec. VI.6]). The latter quantity is independent of the basis chosen and defines a norm making ${\mathcal{L}}_{2}(H)$ a Banach space. It is actually a Hilbert space with the scalar product

[TABLE]

where $\{e_{k}\}_{k\in\mathbb{N}}$ is any orthonormal basis of $H$.

Stochastic processes.

Let $(\Omega,\mathcal{F},(\mathcal{F}_{t})_{t\geq 0},\mathbb{P})$ be a filtered probability space satisfying the usual conditions. Given $p\in[1,+\infty)$, $T>0$, and a Hilbert space $U$, we denote by $\mathcal{M}_{\mathcal{P}}^{p,T}(U)$ the set of all (equivalence classes of) progressively measurable processes $X\colon[0,T]\times\Omega\rightarrow U$ such that

[TABLE]

This is a Banach space with the norm $|\cdot|_{\mathcal{M}_{\mathcal{P}}^{p,T}(U)}$. Next, we denote by $\mathcal{M}_{\mathcal{P}}^{p,loc}(U)$ the space of all (equivalence classes of) progressively measurable processes $X\in\mathcal{M}_{\mathcal{P}}^{p,T}(U)$ such that $X|_{[0,T]\times\Omega}\in\mathcal{M}_{\mathcal{P}}^{p,T}(U)$ for every $T>0$. We denote by $\mathcal{K}_{\mathcal{P}}^{p,T}(U)$ the set of all (equivalence classes of) progressively measurable processes $X\in\mathcal{M}_{\mathcal{P}}^{p,T}(U)$ such that

[TABLE]

is continuous. This is a Banach space with the norm

[TABLE]

Next, we denote by $\mathcal{K}_{\mathcal{P}}^{p,loc}(U)$ the space of all (equivalence classes of) progressively measurable processes $X\colon[0,+\infty)\times\Omega\rightarrow U$ such that $X|_{[0,T]\times\Omega}\in\mathcal{K}_{\mathcal{P}}^{p,T}(U)$ for every $T>0$. We also say that elements of $\mathcal{K}_{\mathcal{P}}^{p,T}(U)$ and $\mathcal{K}_{\mathcal{P}}^{p,loc}(U)$ are “$p$-mean continuous”.

2.2 $G$-derivative

Here we provide the notion of $G$-derivative for functions $f:{H}\rightarrow\mathbb{R}$, where $H$ is a Hilbert space. The latter notion is considered in [16] when $G$ is a map $G:{U}\rightarrow{\mathcal{L}}_{u}(Z,U)$, with $U,Z$ Banach spaces. Here we restrict to the case of constant $G$.

Recall that, if $f:H\rightarrow\mathbb{R}$, the Fréchet derivative of $f$ at $x$ (if it exists) is the (unique) linear functional $Df(x)\in H^{*}\cong H$ such that

[TABLE]

Definition 2.1** ($G$-derivative).**

Let $H,K$ be Hilbert spaces, let $f:{H}\rightarrow\mathbb{R}$ and $G\in{\mathcal{L}}_{u}(K,H)$. We say that $f$ is continuously $G$-Fréchet differentiable at $x\in H$ (briefly, $G$-differentiable at $x\in H$) if there exists $D^{G}f(x)\in K^{*}\cong K$ (clearly, if it exists, then it is unique), called the $G$-derivative of $f$ at $x$, such that

[TABLE]

We denote by $C^{1,G}_{b}(H)$ the space of all maps $f:H\rightarrow\mathbb{R}$ such that $f$ is continuously $G$-differentiable over $H$, i.e. such that $f$ is $G$-differentiable at each $x\in H$ and the map $D^{G}f:H\rightarrow K$ belongs to $C_{b}(H,K)$. In the special case $K=H$ and $G=I$, we simply use the standard notation $C^{1}_{b}(H)$.

Remark 2.2**.**

Note that, in the definition of the $G$-derivative, one considers only the directions in $H$ selected by the range of $G$. When $K=H$ and $G=I$ it reduces to the Fréchet derivative, i.e. $Df=D^{G}f$. Clearly, if $f$ is $G$-differentiable at $x$, then it is also $G$-Gateaux differentiable at $x$, in the sense that

[TABLE]

moreover, the limit above is uniform in $k\in{\mathcal{D}}(G)\cap B_{K}(0,R),$ for every $R>0$. Conversely, if there exists $k^{\prime}\in K$ such that

[TABLE]

then $f$ is $G$-differentiable at $x\in H$ and $D^{G}f(x)=k^{\prime}$.

The notion of $G$-derivative allows to deal with functions which are not Gateaux differentiable, as shown by the following example.

Example 2.3**.**

Let $f:\mathbb{R}^{2}\rightarrow\mathbb{R}$ be defined by $f(x_{1},x_{2})\coloneqq\left|x_{1}\right|x_{2}$. Clearly, $f$ does not admit directional derivative in the direction $(1,0)$ at the point $(x_{1},x_{2})=(0,1)$. On the other hand, if we consider $G\in{\mathcal{L}}(\mathbb{R}^{2})\cong\mathbb{R}^{2}$, defined by $G=(0,1)$, then $f$ admits $G$-Fréchet derivative at every $(x_{1},x_{2})\in\mathbb{R}^{2}$.

Remark 2.4**.**

Clearly, if $f$ is Fréchet differentiable at some $x\in H$ and $G\in{\mathcal{L}}(K,H)$, it turns out that $f$ is $G$-Fréchet differentiable at $x$ and

[TABLE]

Also, if $f$ is both Fréchet differentiable and $G$-differentiable at some $x\in H$, then $Df(x)\in{\mathcal{D}}(G^{*})$ and (2.5) holds true. Indeed, we get by Fréchet differentiability

[TABLE]

On the other hand, by $G$-Fréchet differentiability we also have

[TABLE]

Hence

[TABLE]

It follows what claimed.

If $G$ is unbounded, a function $f:H\rightarrow\mathbb{R}$ may be Fréchet-differentiable at some $x\in H$ and yet not $G$-Fréchet differentiable there, as shown by the following example.

Example 2.5**.**

Let $H,K$ be Hilbert spaces, let $G:{\mathcal{D}}(G)\subsetneq K\rightarrow H$ be a closed densely defined unbounded linear operator on $H$, and let $G^{*}:{\mathcal{D}}(G^{*})\subsetneq H\rightarrow K$ be its adjoint. Next, let $f:U\rightarrow\mathbb{R}$ be defined by $f(x)\coloneqq\frac{1}{2}|x|_{H}^{2}$. Clearly, $f$ is Fréchet differentiable at every $x\in H$ and $Df(x)=x$. On the other hand, if $f$ was also $G$-differentiable at every $x\in H$, by Remark 2.4 it would follow $x\in{\mathcal{D}}(G^{*})$ for every $x\in H$, i.e. ${\mathcal{D}}(G^{*})=H$, a contradiction.

3 Formulation of the stochastic optimal control problem

We are concerned with the optimal control of an Ornstein-Uhlenbeck process valued in a Hilbert space $H$. Precisely, let $H,K,\Xi$ three real separable Hilbert spaces, let $(U,|\cdot|_{U})$ be a real Banach space and let $\Lambda\subseteq U$ be measurable and endowed with the $\sigma$-algebra induced by ${\mathcal{B}}(U)$, the Borel $\sigma$-algebra of $U$. Let $(\Omega,\mathcal{F},\{\mathcal{F}_{t}\}_{t\geq 0},\mathbb{P})$ be a complete filtered probability space satisfying the usual conditions, let $W=(W_{t})_{t\geq 0}$ be a $\Xi$-valued cylindrical Brownian motion (see [10, Ch. 4]), and consider the controlled SDE

[TABLE]

where the control process $u(\cdot)$, taking values in $\Lambda$, belongs to a suitable space of admissible controls and the coefficients $A,G,L,\sigma$ satisfy the following assumptions, which will be standing and not repeated throughout the paper.

Assumption 3.1**.**

2. (i)

$A:{\mathcal{D}}(A)\subseteq H\rightarrow H$* is a closed densely defined linear operator generating a $C_{0}$-semigroup \big{\{}e^{tA}\big{\}}_{t\geq 0} of operators of $\mathcal{L}(H)$.* 3. (ii)

$\sigma\in{\mathcal{L}}(\Xi,H)$,  $e^{sA}\sigma\sigma^{\ast}e^{sA^{\ast}}\in{\mathcal{L}}_{1}(H)$ for all $s>0$, and there exists $\gamma\in(0,1/2)$ such that

[TABLE] 4. (iii)

$G:{\mathcal{D}}(G)\subseteq K\rightarrow H$* is a closed densely defined444The assumption that $G$ is densely defined can be done without loss of generality, as one can always restrict $K$ to $\overline{{\mathcal{D}}(G)}$. linear operator such that $e^{sA}G:{\mathcal{D}}(G)\rightarrow H$ can be extended for every $s>0$ to a continuous linear operator defined on $K$ that we denote by $\overline{e^{sA}G}$. Moreover, there exists $C_{G}>0$, $a_{G}\in\mathbb{R}$ and $\beta\in[0,1)$ such that*

[TABLE] 5. (iv)

$L:\Lambda\rightarrow K$* is measurable and \big{|}L(u)\,\big{|}_{K}\leq C_{L}(1+|u|_{U}) for some $C_{L}>0$.*

Remark 3.2**.**

Since for every $t>0$ and $s\geq 0$ the operators $\overline{e^{(s+t)A}G}$ and $e^{sA}\overline{e^{tA}G}$ belong to ${\mathcal{L}}(K,H)$ and coincide on the dense subset ${\mathcal{D}}(G)\subseteq K$, we have

[TABLE]

This implies that the map $(0,+\infty)\rightarrow\mathcal{L}(K,H),\ s\mapsto\overline{e^{sA}G}$ is strongly continuous, i.e. $s\mapsto\overline{e^{sA}G}x$ is continuous for each $x\in H$.

We now take

[TABLE]

which will be fixed in the rest of the paper. We consider, as space of admissible controls, the space of processes

[TABLE]

The reason for the choice of $\beta$ in (3.2) and of $p$ in (3.4)-(3.5) relies on the following result (cf. also [20, Prop. 8.8] and [23, Lemma 3.2]), which will guarantee well-posedness of the controlled state equation (Proposition 3.4).

Lemma 3.3**.**

Let $E,V$ be real Banach spaces, let $\beta\in[0,1)$, $p>\frac{1}{1-\beta}$. Let $f\in L_{loc}^{p}([0,+\infty);E)$ and let $g:(0,+\infty)\rightarrow\mathcal{L}(E,V)$ be strongly continuous555Meaning that $g(\cdot)e:(0,+\infty)\rightarrow V$ is continuous for each $e\in E$. and such that $|g(s)|_{\mathcal{L}(E,V)}\leq{C_{0}(s^{-\beta}\vee 1)}$ for some $C_{0}>0$ for every $s\in(0,+\infty)$. Then $F:\mathbb{R}^{+}\rightarrow V$ defined as Bochner integral by

[TABLE]

is well defined and continuous.

Proof.

Let $t>0$. First of all, we note that the map

[TABLE]

is measurable for each $t>0$. Indeed, given $t>0$ the above map can be seen as the composition $h_{1}\circ h_{2}$ where

[TABLE]

Now, $h_{2}$ is clearly measurable. Also $h_{1}$ is measurable, as it is continuous: indeed $g(\cdot)e$ is continuous for each $e\in E$ and $\{g(s)\}_{s\in[\varepsilon,t]}\subseteq\mathcal{L}(E,V)$ is a family of uniformly bounded operators for each $\varepsilon\in(0,t)$. Hence $h_{1}\circ h_{2}$ is measurable.

Given the above, it makes sense to consider $\int_{0}^{t}g(t-s)f(s)ds$ in Bochner sense for each $t>0$. By Hölder’s inequality, setting $\kappa:=-\frac{\beta p}{p-1}+1>0$, we have for each $t>0$

[TABLE]

This show, at once, that $F$ is well defined as Bochner integral in $V$ and that $\lim_{t\rightarrow 0^{+}}F(t)=0$, so $F$ is continuous at [math].

Let us show now that $F$ is continuous on each interval of the form $\left[{t_{0}},T\right]$ with $t_{0}\in(0,T)$. Set, for $\varepsilon\in(0,{t_{0}})$,

[TABLE]

By dominated convergence we easily see that $F_{\varepsilon}$ is continuous on $\left[{t_{0}},T\right]$. Moreover, using again Hölder’s inequality we have, for all $t\in[t_{0},T]$

[TABLE]

This show $F_{\varepsilon}\rightarrow F$ uniformly in $[t_{0},T]$, hence $F$ is continuous in $[t_{0},T]$, concluding the proof.  ${\square}$

Proposition 3.4**.**

For each $u(\cdot)\in{\mathcal{U}}_{p}$, the process

[TABLE]

is well-defined and belongs to ${\mathcal{K}}_{\mathcal{P}}^{1,loc}(H)$. Moreover, it admits a version with continuous trajectories.

Proof.

By Remark 3.2 and Assumption 3.1(iii)-(iv), we can apply Lemma 3.3 with

[TABLE]

and $g(s)\in\mathcal{L}(E,V)$ defined by

[TABLE]

It follows that

[TABLE]

is well defined as stochastic process and belongs to ${\mathcal{K}}_{\mathcal{P}}^{1,loc}(H)$. We can repeat the argument employed above dealing now with trajectories. Fixing $\omega\in\Omega$ and applying Lemma 3.3 with

[TABLE]

it follows that the map

[TABLE]

is continuous. The latter integral expression, for varying $\omega\in\Omega$, clearly provides a version of (3.7) with continuous trajectories.

On the other hand, in view of Assumption 3.1(ii), from [10, Th. 5.2 and Th. 5.11] we know that the stochastic convolution

[TABLE]

is a (well defined) stochastic process belonging to ${\mathcal{K}}_{\mathcal{P}}^{2,loc}(H)$ and admitting a version with continuous trajectories, concluding the proof.  ${\square}$

We refer to the process (3.6) as the controlled Ornstein-Uhlenbeck process or mild solution of SDE (3.1). We always consider its version (unique, up to indistinguishability) with continuous trajectories.

Let $\lambda>0$, $x\in H$, and let $l:H\times\Lambda\rightarrow\mathbb{R}$ be such that

[TABLE]

Consider the functional

[TABLE]

By (3.8), the functional above is well defined (possibly with value $+\infty$) for all $x\in H$ and $u(\cdot)\in{\mathcal{U}}_{p}$. The stochastic optimal control problem consists in minimizing the functional over the set of admissible controls ${\mathcal{U}}_{p}$, i.e. in solving the optimization problem

[TABLE]

The function $V:H\rightarrow\mathbb{R}\cup\{+\infty\}$ is the so called value function of the optimization problem. If $x\in H$ is such that $V(x)<\infty$ and $u^{*}(\cdot)$ is such that $V(x)=J(x;u^{*}(\cdot))$, then $u^{*}(\cdot)$ is called optimal strategy and the associated state trajectory is called optimal state; moreover the couple \big{(}u^{*}(\cdot),X(\cdot;x,u^{*}(\cdot))\big{)} is called an optimal couple.

4 Generalized Dynkin’s formula

The aim of the present section is to prove an abstract Dynkin formula for the controlled Ornstein-Uhlenbeck process (3.6) composed with suitably smooth functions $\varphi:H\rightarrow\mathbb{R}$.

4.1 Transition semigroups, generators and $G$-derivatives

We consider the family of transition semigroups associated to the uncontrolled version of (3.6) and to the same process under constant controls. Precisely, we denote by $X^{(k)}(\cdot;x)$, where $k\in K$, the Ornstein-Uhlenbeck process starting at $x\in H$ with extra drift $Gk$; i.e., the mild solution to

[TABLE]

Its explicit expression is

[TABLE]

Correspondingly, we define the family of linear operators \big{\{}P^{(k)}_{t}\big{\}}_{t\geq 0} in the space $C_{b}(H)$ as

[TABLE]

In Proposition 4.3(i) below we will show that the family \big{\{}P^{(k)}_{t}\big{\}}_{t\geq 0} is a one-parameter semigroup of linear operators in the space $C_{b}(H)$. According to the related the literature, we call it the transition semigroup associated to the process $X^{(k)}$. Unfortunately, such semigroup is not in general a $C_{0}$-semigroup in $C_{b}(H)$, not even in the case $k=0$. Indeed, in the framework of spaces of functions not vanishing at infinity, the $C_{0}$-property, i.e. the fact that $\lim_{s\rightarrow 0^{+}}P^{(k)}_{s}\varphi=\varphi$ in the sup norm for every $\varphi$, fails even in basic cases. For instance, this property fails in the case of the Ornstein-Uhlenbeck semigroup in the space $C_{b}(\mathbb{R})$ (see, e.g., [4, Example 6.1] for a counterexample in $UC_{b}(\mathbb{R})$, or [8, Lemma 3.2], which implies this is a $C_{0}$-semigroup in $UC_{b}(\mathbb{R})$ if and only if the drift of the SDE vanishes). Even worse: given $\varphi\in C_{b}(H)$, the map $[0,+\infty)\rightarrow C_{b}(H)$, $t\mapsto P_{t}^{(k)}\varphi$ is not in general measurable, as shown in [16, Example 4.5]. This prevents, for instance, to intend in Bochner sense, in the space $C_{b}(H)$ for each $g\in C_{b}(H)$, the integral defining the Laplace transform

[TABLE]

Nevertheless, one can get, in a weaker sense, several statements of the classical theory of $C_{0}$-semigroups. This is performed, e.g., by the theory of ${\mathcal{K}}$-semigroups (introduced in [4], see also [6], with the different terminology of weakly continuous semigroups) and $\pi$-semigroups (introduced in [43, 44]). Both theories (a survey of which can be found in Appendix B.5 of [14]) can be applied here getting substantially the same results. We employ the $\pi$-semigroups approach, as it seems more natural in our context. The definition of $\pi$-convergence can be found e.g. in [12, p. 111], where it is called bp-convergence (bounded-pointwise convergence) and in [43, 44]; the former in the space $C_{b}(H)$, the latter in the space $UC_{b}(H)$.

Definition 4.1** ($\pi$-convergence).**

A sequence of functions $(f_{n})\subseteq C_{b}(H)$ is said to be $\pi$-convergent to a function $f\in C_{b}(H)$ if

[TABLE]

*Such convergence is denoted by $f_{n}\xrightarrow{\pi}f$ or by $f=\pi\mbox{-}\!\!\lim_{n\rightarrow\infty}f_{n}.$ *

Now we recall the definition of $\pi\mbox{-}$semigroup as given in [43, 44]. Here we state it in the space of continuous and bounded functions (the aforementioned references deal with the space of uniformly continuous and bounded functions, but also explain how to extend the definition to $C_{b}(H)$).

Definition 4.2**.**

A semigroup \big{\{}P_{t}\big{\}}_{t\geq 0} of bounded linear operators on $C_{b}(H)$ is called a $\pi\mbox{-}$semigroup on $C_{b}(H)$ if it satisfies the following conditions.

1. (P1)

There exist $M\geq 1$ and $\alpha\in\mathbb{R}$ such that $|P_{t}[f]|_{\infty}\leq Me^{\alpha t}|f|_{\infty}$ for every $t\in\mathbb{R}^{+}$, $f\in C_{b}(H)$. 2. (P2)

For each $x\in H$ and $f\in C_{b}(H)$, the map $\mathbb{R}^{+}\rightarrow\mathbb{R},\ t\mapsto P_{t}[f](x)$ is continuous. 3. (P3)

We have

[TABLE]

Define

[TABLE]

and

[TABLE]

It is proved (see [6, Lemma. 5.7] combined with the discussion of [43, Sec. 4.3]) that, for $\varphi$ sufficiently smooth,

[TABLE]

We will use (4.7) to formally motivate the definition of mild solution (Definition 5.1) of the HJB equation associated to the control problem of Section 3.

Proposition 4.3**.**

Let $k\in K$.

1. (i)

*The family of linear operators \big{\{}P_{t}^{(k)}\big{\}}_{t\geq 0} defined in (4.3) is a $\pi$-semigroup on $C_{b}(H)$. We denote by ${\mathcal{A}}^{(k)}$ its infinitesimal generator. * 2. (ii)

The operator

[TABLE]

belongs to ${\mathcal{L}}(C_{b}(H))$ for every $\lambda>0$ and is the resolvent of ${\mathcal{A}}^{(k)}$:

[TABLE] 3. (iii)

We have777At $t=0$ the derivative is intended as right derivative.**

[TABLE]

Proof.

Claims (ii)-(iii) follow from [43, Prop. 3.2, Prop. 3.6] or [44, Prop. 6.2.7, Prop. 6.2.11](888These references deal mainly in the space of uniformly continuous and bounded functions — we warn that the author denotes by $C_{b}(H)$ the latter space. The extension to the space of continuous and bounded function — our space $C_{b}(H)$ — is illustrated in [43, Sec. 5] and [44, Sec. 6.5].) once one proves claim (i), which we prove below.

Proof of (i). First of all, we prove that \big{\{}P_{t}^{(k)}\big{\}}_{t\geq 0} is a semigroup of linear operators on $C_{b}(H)$. The fact that $P_{0}^{(k)}=I$ and that $P_{t}^{(k)}\in{\mathcal{L}}(C_{b}(H))$ for all $t\geq 0$ is immediate. The semigroup property of \big{\{}e^{tA}\big{\}}_{t\geq 0} and (3.3) yield

[TABLE]

The latter shows the strong Markov property of $X^{(k)}$ and then the fact that \big{\{}P^{(k)}_{t}\big{\}}_{t\geq 0} satisfies the semigroup property follows as consequence (see, e.g., [10, Cor. 9.15]). Now we show the other properties of Definition 4.2. (P1) is obviously verified with $M=1$ and $\alpha=0$. (P2) of Definition 4.2 corresponds to

[TABLE]

The latter follows from continuity of trajectories of $X^{(k)}(\cdot;x)$ and dominated convergence. Finally, (P3) of Definition 4.2 is verified by dominated convergence.  ${\square}$

A key step towards the main goal of this section, i.e. the proof of a generalized Dynkin formula for $\varphi(X(\cdot;x,u(\cdot))$ with a suitably regular $\varphi$, consists in showing the following decomposition of $\mathcal{A}^{(k)}$ when acting on the function $\varphi$

[TABLE]

Looking at \big{\{}P_{t}^{(k)}\big{\}}_{t\geq 0} as to a perturbation of \big{\{}P_{t}^{(0)}\big{\}}_{t\geq 0}, (4.9) is obtained in [25, Theorem 5.2] in the context of $C_{0}$-semigroups with respect to mixed topology of $C_{b}(H)$ and in [15, Theorem 4.6] in the context of bi-continuous semigroups. However, these references would require the assumptions that $\varphi\in C^{1}_{b}(H)$ and $A,\sigma$ are such that $C^{1}_{b}(H)\subseteq\mathcal{D}(\mathcal{A}^{(0)})$ and $G\in{\mathcal{L}}(H)$. This would allow, in particular, to write the term $\langle D^{G}\varphi(\cdot),k\rangle_{K}$ in the formula above as $\langle D\varphi(\cdot),Gk\rangle_{H}$, simplifying a lot the framework. Here we need to be sharper in this respect in order to cover other cases of interest in applications, e.g., the case of unbounded $G$, occurring in boundary control problems. To this purpose we introduce the class of functions

[TABLE]

Our generalized Dynkin formula will hold for functions belonging to $\mathcal{D}({\mathcal{A}}^{(0)})\cap\mathcal{S}^{A,G}(H)$. In Appendix 6 we provide sufficient conditions on $A,G,\varphi$ ensuring that $\varphi\in\mathcal{S}^{A,G}(H)$.

Proposition 4.4**.**

*Let $\varphi\in\mathcal{D}({\mathcal{A}}^{(0)})\cap\mathcal{S}^{A,G}(H)$. Then (4.9) holds. *

Proof.

Since $\varphi\in{\mathcal{D}}({\mathcal{A}}^{(0)})$, we can write for every $x\in H$

[TABLE]

if the last limit exists. Observe that

[TABLE]

Therefore, since $\varphi\in\mathcal{S}^{A,G}(H)$, continuity of $t\mapsto\int_{0}^{t}\overline{e^{sA}G}k\,ds$ and by dominated convergence yield

[TABLE]

The claim follows.  ${\square}$

4.2 Proof of the generalized Dynkin’s formula

We introduce the linear space ${\mathcal{K}}^{s,p}$ of $K$-valued $p$-integrable càdlàg simple processes. An element $\kappa(\cdot)\in{\mathcal{K}}^{s,p}$ is of the form

[TABLE]

for some $n\in\mathbb{N}$, $0={t_{0}}<t_{1}<...<t_{n}=+\infty,$ and $\{k_{i}\}_{i=0,...,{n-1}}$ such that $k_{i}\in L^{p}(\Omega,{\mathcal{F}}_{t_{i}},\mathbb{P};K)$ for all $i=0,...,{n-1}$. Processes in ${\mathcal{K}}^{s,p}$ are progressively measurable. By arguing as in the proof of Proposition 3.4 we get that, for any $\kappa(\cdot)\in{\mathcal{K}}^{s,p}$, the process

[TABLE]

is well defined, belongs to ${\mathcal{M}}^{1,loc}_{\mathcal{P}}(H)$ and has a version with continuous trajectories. We will always refer to the version of this process (unique up to indistinguishability) having continuous trajectories. Given $\kappa(\cdot)\in{\mathcal{K}}^{s,p}$, we write

[TABLE]

Again arguing as in the proof of Proposition 3.4 we see that this process has a version with having continuous trajectories. As above we will always refer to this version (unique up to indistinguishability).

Recall that, if $V_{1}$, $V_{2}$ are two random variables with values, respectively, in two measurable spaces $(E_{1},{\mathcal{E}}_{1})$ and $(E_{2},{\mathcal{E}}_{2})$, a version of the conditional law of $V_{1}$ given $V_{2}$ is a family of probability measures \big{\{}\mu(\cdot,v_{2})\big{\}}_{v_{2}\in E_{2}} on $(E_{1},{\mathcal{E}}_{1})$ such that, for every $f\in B_{b}(E_{1}\times E_{2};\mathbb{R})$, the map $v_{2}\mapsto\int_{E_{1}}f(v_{1},v_{2})\mu(dv_{1},v_{2})$ is measurable and

[TABLE]

where $\nu=\textsl{Law}\,(V_{2})$. This family, if it exists, is unique up to $\nu$-null measure sets.

Lemma 4.5**.**

Let $\kappa(\cdot)\in{\mathcal{K}}^{s,p}$ be in the form (4.11) and $t\in[t_{i-1},t_{i})$ for some $i=1,...,n.$ A version of the conditional law of $X^{\kappa(\cdot)}(t;x)$ given the couple $(X^{\kappa(\cdot)}(t_{i-1};x),k_{i-1})$ is the family

[TABLE]

Proof.

The proof is standard (see [36, Ch. 2, Sec. 9] in finite dimension and in a much more general setting) and we omit it for brevity.  ${\square}$

Lemma 4.6**.**

Let $\varphi\in\mathcal{D}({\mathcal{A}}^{(0)})\cap\mathcal{S}^{A,G}(H)$ and $\kappa(\cdot)\in{\mathcal{K}}^{s,p}$. Then

[TABLE]

where the derivative has to be intended as right derivative at the times $\{t_{1},...,t_{n}\}$, where the simple process $\kappa(\cdot)$ jumps.

Proof.

Let $\kappa(\cdot)\in{\mathcal{K}}^{s,p}$ be as in (4.11), $t\in[t_{i-1},t_{i})$ for some $i=1,...,n$, and $\varphi\in{\mathcal{D}}({\mathcal{A}}^{(0)})\cap C^{1,G}_{b}(H)$. Denote by $\nu$ the law of the couple $(X^{\kappa(\cdot)}(t_{i-1};x),k_{i-1})$. By Lemma 4.5, we have

[TABLE]

Now we differentiate under the integral sign using the fact that, by Proposition 4.4, $\varphi\in{\mathcal{D}}({\mathcal{A}}^{(k^{\prime})})$ and the fact that $(t,x^{\prime})\mapsto P_{t-t_{i-1}}^{(k^{\prime})}[{\mathcal{A}}^{(k^{\prime})}[\varphi]](x^{\prime})$ is bounded over $[t_{i-1},t_{i})\times H$. Then, using Proposition 4.3(i) and (4.9), we get

[TABLE]

the claim.  ${\square}$

Lemma 4.7**.**

For each $u(\cdot)\in{\mathcal{U}}_{p}$ and $T>0$, there exists a sequence $\{\kappa_{n}\}_{n\in\mathbb{N}}\subset{\mathcal{K}}^{s,p}$ such that

[TABLE]

Proof.

Fix $T>0$ and set $\kappa(\cdot):=L(u(\cdot))$. By standard arguments (see, e.g., [34, Ch. III, Lemma. 2.4, p.132])999It is worth to point out some differences. First, we are dealing with càdlàg approximations (as it is more meaningful and natural to state Proposition 4.6) rather than with càglàd (as in [34, Ch. III, Lemma. 2.4, p.132]): this is not a problem as, from the point of view of integration, these classes coincide. Second, we are dealing with Hilbert-valued processes: therefore, more technical care is needed as the approximation is produced by Bochner integration., we can construct a sequence $\{\kappa_{n}\}_{n\in\mathbb{N}}\subset{\mathcal{K}}^{s,p}$ such that

[TABLE]

Then, using the expression (3.6) for the state variable, the convergence

[TABLE]

follows by simply applying dominated convergence.  ${\square}$

Theorem 4.8** (Dynkin’s formula).**

Let $\varphi\in\mathcal{D}({\mathcal{A}}^{(0)})\cap\mathcal{S}^{A,G}(H)$. Then, for every $\lambda>0$, $T>0$, and $u(\cdot)\in{\mathcal{U}}_{p}$, we have

[TABLE]

Proof.

Let $u(\cdot)\in{\mathcal{U}}_{p}$ and take the approximating sequence $\left\{\kappa_{n}\right\}_{n\in\mathbb{N}}$ provided by Lemma 4.7. Then, applying, for each $n\in\mathbb{N}$, Lemma 4.6, we obtain from (4.13) (by taking the right derivatives at $t_{i}$), for all $t\geq 0$ and $\lambda>0$,

[TABLE]

Since the function t\mapsto\mathbb{E}\left[e^{-\lambda t}\varphi\big{(}X^{\kappa_{n}(\cdot)}(t;x)\big{)}\right] is everywhere continuous and stepwise differentiable, we can apply the Fundamental Theorem of Calculus. So, integrating on $[0,T]$, we get

[TABLE]

Now, letting $n\rightarrow+\infty$, we get the claim by dominated convergence from Lemma 4.7, observing that $\varphi$, $D^{G}\varphi$, and ${\mathcal{A}}^{(0)}[\varphi]$ are bounded.  ${\square}$

Remark 4.9**.**

The results of this section, in particular Theorem 4.8, can be extended, at the price of straightforward technical complications, to the case when the basic space of functions is, instead of $C_{b}(H)$, the space $C_{m}(H)$, where $m>0$, used e.g. in [16]:

[TABLE]

Also the results of next Section 5 can be extended to this setting covering more general cases, in particular when the current cost of the control problem has polynomial growth in $x$. We do not do this here for brevity.

5 HJB equation, verification theorem and optimal feedbacks

By standard Dynamic Programming arguments, one formally associates to the control problem of Section 3 the following HJB equation for the value function (3.10):

[TABLE]

where $Q=\sigma\sigma^{*}$ and the Hamiltonian $F$ is defined by

[TABLE]

where

[TABLE]

Note that this definition is only formal as $GL(u)$ may be not defined, since $L(u)$ may not belong to $\mathcal{D}(G)$. It is then convenient to introduce the modified Hamiltonian

[TABLE]

where

[TABLE]

Observing that

[TABLE]

(5.1) can be formally rewritten as

[TABLE]

Note that, in principle, $F_{0}$ may take the value $-\infty$ somewhere. The concept of mild solution to (5.1) relies on Proposition 4.3(ii) and on (4.7), inspiring an integral form of (5.6) through the use of the semigroup \big{\{}P_{s}^{(0)}\big{\}}_{s\geq 0}.

Definition 5.1**.**

We say that a function $v:H\rightarrow\mathbb{R}$ is a mild solution to (5.6) if it belongs to $C_{b}^{1,G}\left(H\right)$, $F_{0}(\cdot,D^{G}v\left(\cdot\right))$ is bounded and $v$ solves the integral equation

[TABLE]

Remark 5.2**.**

The problem of existence and uniqueness of mild solutions for equations in the form (5.6) is addressed in [16] and in [14, Ch. 4]. In particular, existence and uniqueness of mild solutions is stated for sufficiently large $\lambda>0$, under the following assumptions (see [16, Cor. 4.12, Th. 3.8(ii)] with $m=0$):

• (A1)

$\overline{e^{tA}G}(K)\subseteq Q_{t}^{1/2}(H)$* for every $t>0$, where $Q_{t}:=\displaystyle\int_{0}^{t}e^{sA}\sigma\sigma^{\ast}e^{sA^{\ast}}ds$.*

• (A2)

The operators101010Here $Q_{t}^{-1/2}$ is the pseudo-inverse of $Q_{t}^{1/2}$.**

[TABLE]

which are well defined by (A1) and bounded by the closed graph theorem, are such that the map $t\mapsto|\Gamma_{G}(t)|_{{\mathcal{L}}(K,H)}$ belongs to $L^{1}_{loc}([0,+\infty),\mathbb{R})$ and is bounded in a neighborhood of $+\infty$.

• (A3)

The Hamiltonian $F_{0}$ satisfies, for suitable $C_{F_{0}}>0$.

[TABLE]

[TABLE]

Some results in the case of locally Lipschitz Hamiltonian are available, up to now, only in special cases (see [10, Sec. 13.3.1] and [5]).

Due to Proposition 4.3(ii), a mild solution $v$ of (5.1) enjoys the property of being a solution to the same equation also in a differential abstract way, i.e., we have the following.

Proposition 5.3**.**

Let $v$ be a mild solution to (5.6). Then $v\in{\mathcal{D}}({\mathcal{A}}^{(0)})$ and

[TABLE]

Proof.

Using Proposition 4.3(ii), we rewrite (5.7) as

[TABLE]

This entails $v\in{\mathcal{D}}({\mathcal{A}}^{(0)})$ and, applying $\lambda-{\mathcal{A}}^{(0)}$ to both sides, we see that $v$ solves (5.8).  ${\square}$

Remark 5.4**.**

By Proposition 5.3 a mild solution $v$ to (5.6) belongs to $\mathcal{D}({\mathcal{A}}^{(0)})$. Hence, in order to apply Theorem 4.8 to it, we only need to assume that $v\in\mathcal{S}^{A,G}(H)$. This is what we indeed assume in all the next results of this section.

5.1 Verification theorem

The proof of the verification theorem relies in the so called fundamental identity.

Proposition 5.5** (Fundamental identity).**

Let (3.8) hold. Let $v$ be a mild solution to (5.6) and assume that $v\in\mathcal{S}^{A,G}(H)$. Let $x\in H$ and let $u(\cdot)\in{\mathcal{U}}_{p}$ be such that

[TABLE]

Then

[TABLE]

Proof.

Let $x\in H$, $T>0$, and let $u(\cdot)\in{\mathcal{U}}_{p}$ be such that (5.10) holds. Using Proposition 5.3 and applying the abstract Dynkin formula (Theorem 4.8) to $t\mapsto e^{-\lambda t}v(X(t;x,u(\cdot)))$, we get

[TABLE]

Since $l$ is measurable and bounded from below by (3.8), the term \mathbb{E}\big{[}\int_{0}^{T}e^{-\lambda t}l\big{(}X(t;x,u(\cdot)),u(t)\big{)}\,dt\big{]} is well defined, possibly equal to $+\infty$. However, (5.10) actually entails

[TABLE]

Then, we can add and subtract \mathbb{E}\big{[}\int_{0}^{T}e^{-\lambda t}l\big{(}X(t;x,u(\cdot)),u(t)\big{)}dt\big{]} in (5.12) and use (5.5) to get, rearranging the terms,

[TABLE]

Now we let $T\rightarrow+\infty$. The right hand side has a limit (possibly $+\infty$), as the integrand is positive. The left hand side clearly converges to $J(x;u(\cdot))-v(x)$. This implies that also the limit of the right hand side is finite and

[TABLE]

The claim follows rearranging the terms.  ${\square}$

Theorem 5.6** (Verification theorem).**

Let (3.8) hold. Let $v$ be a mild solution to (5.6) and assume that $v\in\mathcal{S}^{A,G}(H)$. We have the following.

• (i)

$v\leq V$* over $H$.*

• (ii)

Let $x\in H$ and assume that there exists $u^{*}(\cdot)\in{\mathcal{U}}_{p}$ such that $\mathbb{P}\times dt-\mbox{a.e.}$

[TABLE]

Then $v(x)=V(x)=J(x;u^{*}(\cdot))$.

Proof.

(i) By (5.11), for all $u(\cdot)\in{\mathcal{U}}_{p}$ such that (5.10) holds, we have $v(x)\leq J(x;u(\cdot))$, which yields this claim.

(ii) Let $u^{*}(\cdot)$ such that (5.14) holds. If $J(x;u^{*}(\cdot))<+\infty$, then, from (5.11), we immediately get $v(x)=J(x;u^{*}(\cdot))$, which, combined with item (i), yields the claim. We now prove that it cannot be $J(x;u^{*}(\cdot))=+\infty$. Assume, by contradiction, that $J(x;u^{*}(\cdot))=+\infty$. Then, by (5.14), we have $\mathbb{P}\times dt-\mbox{a.e.}$

[TABLE]

By (5.8), $F_{0}(\cdot,D^{G}v(\cdot))$ is bounded. Hence, Assumption 3.1-(iv), the fact that $u^{*}(\cdot)\in{\mathcal{U}}_{p}$ and (5.15) imply \mathbb{E}\big{[}\int_{0}^{T}e^{-\lambda t}l\big{(}X(t;x,u^{*}(\cdot)),u^{*}(t)\big{)}\,dt\big{]}<\infty for all $T>0$. Then, we can argue as in the proof of Proposition 5.5 getting (5.13) with $u^{*}(\cdot)$ in this case and, using again (5.14),

[TABLE]

Letting $T\rightarrow+\infty$ we get $v(x)=J(x;u^{*}(\cdot))=+\infty$, a contradiction, as $v$ is finite.  ${\square}$

5.2 Optimal feedback controls

As usual, the verification theorem is composed of two statements: the first one states that the solution to the HJB equation enjoys the property of being smaller than the value function; the second one is the most important from the point of view of the control problem, as it furnishes a sufficient condition of optimality ((5.14) in our case). Then, the problem becomes the so-called synthesis of an optimal control, i.e. to produce a control $u^{*}(\cdot)$ verifying such condition. The answer relies in the study of the closed loop equation.

Let $v$ be a mild solution to HJB equation (5.6). Assuming that the infimum of the map

[TABLE]

is attained and defining the multivalued function (feedback map)

[TABLE]

the closed loop equation (CLE) associated with our problem and to $v$ is indeed a stochastic differential inclusion:

[TABLE]

We have the following result.

Corollary 5.7**.**

Let (3.8) hold. Let $v$ be a mild solution to (5.6) and assume that $v\in\mathcal{S}^{A,G}(H)$. Let $x\in H$ and assume that the feedback map $\Phi$ defined in (5.17) admits a measurable selection $\phi:H\rightarrow U$ and consider the SDE

[TABLE]

Assume that (5.19) has a mild solution in $\mathcal{M}_{\mathcal{P}}^{1,loc}(U)$, i.e. there exists $X_{\phi}(s;x)\in\mathcal{M}_{\mathcal{P}}^{1,loc}(U)$ such that

[TABLE]

Define, for $s\geq 0$, $u_{\phi}(s):=\phi(X_{\phi}(s;x))$ and assume that $u_{\phi}(\cdot)\in{\mathcal{U}}_{p}$. Then $v(x)=V(x)=J(x;u_{\phi}(\cdot))$. In particular the couple $({u_{\phi}}(\cdot),{X_{\phi}}(\cdot;x))$ is optimal at $x$.

*Moreover, if $\Phi(x)$ is single-valued and the mild solution to (5.19) is unique, then the optimal control is unique. *

Proof.

Consider the couple $(u_{\phi}(\cdot),X_{\phi}(\cdot))$ and observe that $X_{\phi}(\cdot)$ is the unique mild solution (in the strong probabilistic sense) of the state equation associated to the control $u_{\phi}(\cdot)$, so that $X_{\phi}(\cdot;x)\equiv X(\cdot;x,u_{\phi}(\cdot))$. By construction such couple satisfies (5.14). Then, by Theorem 5.6-(ii) we obtain that it is optimal.

Let us address now the uniqueness issue. We observe that, if $(\hat{u}(\cdot),X(\cdot;x,\hat{u}(\cdot)))$ is another optimal couple at $x$, we immediately have, by (5.11) and the fact that $v(x)=V(x)$,

[TABLE]

As the integrand is always negative and as $\Phi$ is single-valued, this implies that $\mathbb{P}\times ds$-a.e. we have \hat{u}(\cdot)=\Phi\big{(}X(\cdot;x,\hat{u}(\cdot))\big{)}. This shows that $X(\cdot;x,\hat{u}(\cdot))$ solves (5.19). Then uniqueness of mild solutions to (5.19) gives the claim.  ${\square}$

We conclude the section commenting on the extension of our results to the case when the control problem is considered in the so-called weak formulation. So far, we have considered our family of stochastic optimal control problems in the strong formulation. It is possible to consider the problem also in the so-called weak formulation, i.e. letting the filtered probability space and the Wiener process vary with the control strategy $u(\cdot)$ (see, e.g., [48, Ch. 2]). More precisely, in the weak formulation, the control strategy is a $6$-tuple $\left(\overline{\Omega},\overline{\mathcal{F}},\{\overline{\mathcal{F}_{t}}\}_{t\geq 0},\overline{\mathbb{P}},\overline{W},\overline{u}(\cdot)\right)$. Calling $\overline{\mathcal{U}}_{p}$ the set such control strategies, the objective is to minimize the cost (3.9) over $\overline{\mathcal{U}}_{p}$. The resulting value function $\overline{V}$ is, in principle, smaller than $V$. The main advantage in choosing such formulation is that existence of optimal control strategies in feedback form is easier to obtain. The verification theorem above also holds when we consider the control problem in its weak formulation. Indeed, the proof of Theorem 5.6 works for every filtered probability space and any cylindrical Brownian motion on it. Hence, letting the filtered probability space and the cylindrical Brownian motion vary, one gets that $v\leq\overline{V}$ over $H$. Moreover, if (5.14) holds for a given control strategy (111111Elements of $\overline{\mathcal{U}}_{p}$ are, rigorously speaking, $6$-tuples; however, for simplicity, we denote them simply by $\overline{u(\cdot)}$.) $\overline{u^{*}}(\cdot)\in\overline{\mathcal{U}}_{p}$, then we have $v(x)=\overline{V}(x)=J(x;\overline{u^{*}}(\cdot))$. One gets the following.

Corollary 5.8**.**

Let (3.8) hold. Let $v$ be a mild solution to (5.6) and assume that $v\in\mathcal{S}^{A,G}(H)$. Let $x\in H$ and assume that the feedback map $\Phi$ defined in (5.17) admits a measurable selection $\phi:H\rightarrow U$. Assume now that (5.19) has a martingale solution121212Weak-mild solution in the terminology of [14]. (see [10, p. 220] or [23, Def. 3.1, p. 75] for the definition) $\overline{X_{\phi}}(\cdot;x)$ in some filtered probability space \left(\overline{\Omega},\overline{\mathcal{F}},\big{\{}\overline{\mathcal{F}_{t}}\big{\}}_{t\geq 0},\,\overline{\mathbb{P}}\right) and for some $\Xi$-valued cylindrical Brownian motion $\overline{W}$ defined on it. Define, for $s\geq 0$, \overline{u_{\phi}}(s)=\phi\big{(}\overline{X_{\phi}}(s;x)\big{)} and assume $\overline{u_{\phi}}(\cdot)\in\overline{\mathcal{U}}_{p}$ (131313In the sense that the $6$-tuple identified by $u_{\phi}$ belongs to $\overline{\mathcal{U}}_{p}$.). Then $v(x)=\overline{V}(x)=J(x;\overline{u_{\phi}}(\cdot))$. In particular \big{(}\overline{u_{\phi}}(\cdot),\overline{X_{\phi}}(\cdot;x)\big{)} is an optimal couple.

6 Applications

In the present section we provide two examples of application of our results.

The first example, fully developed, concerns the optimal control of the stochastic heat equation in a given space region ${\mathcal{O}}\subseteq\mathbb{R}^{d}$ when the control can be exercised only at the boundary $\partial{\mathcal{O}}$. Precisely, we consider the case when the control at the boundary enters through a Neumann-type boundary condition, corresponding to control the heat flow at the boundary. The existence and uniqueness of mild solutions to the associated elliptic HJB equation in this case is guaranteed (under suitable conditions) by the results of [16].

The second example concerns the optimal control of a stochastic differential equation with delay in the control process (see [29, 30] for the treatment of the same problem over finite horizon). In this case, the result we give needs to assume the existence of a mild solution to the associated elliptic HJB equation. The reason for that is that a theory of mild solutions for elliptic HJB equations associated to this kind problem has not been yet developed in the elliptic case. Indeed, unlike the first example, this kind of equations is not covered by the results of [16], due to the lack of $G$-smoothing. In this case it is needed an ad hoc treatment of the equation, dealing with the specific case at hand, to show the existence of mild solutions (see, e.g., the aforementioned references [29, 30] in the parabolic case). Although a result of this kind for elliptic equation seems straightforward, a rigorous statement of this result has not been rigourously fixed yet. For this reason, we limit ourselves to provide a weaker result taking the existence of mild solutions to the associated HJB equation as an assumption and leaving the investigation of that for future work. Due to the lack of a rigourous background on which relying our results, we do not state in this case a theorem and just keep the arguments at the level of an informal exposition.

6.1 Neumann Boundary control of a stochastic heat equation with additive noise

We consider the optimal control of a nonlinear stochastic heat equation in a given space region ${\mathcal{O}}\subseteq\mathbb{R}^{d}$ when the control can be exercised only at the boundary of ${\mathcal{O}}$.

6.1.1 Problem setup

Let ${\mathcal{O}}$ be an open, connected, bounded subset of ${\mathbb{R}}^{d}$ with regular (in the sense of [37, Sec. 6]) boundary $\partial{\mathcal{O}}$141414We stress that such conditions may allow corners in the boundary: in particular, when $d=2$ squares satisfy the required regularity.. We consider the controlled dynamical system driven by the following SPDE in the time interval $[0,+\infty)$:

[TABLE]

where:

• •

$y:[0,+\infty)\times\mathcal{O}\times\Omega\rightarrow\mathbb{R}$ is the stochastic process describing the evolution of the temperature distribution and is the state variable of the system;

• •

${\gamma_{0}}:[0,+\infty)\times\partial\mathcal{O}\times\Omega\rightarrow\mathbb{R}$ is the stochastic process representing the heat flow at the boundary; it is the control variable of the system and acts at the boundary of it: this is the reason of the terminology “boundary control";

• •

$n$ is the outward unit normal vector at the boundary $\partial{\mathcal{O}}$;

• •

$x\in L^{2}(\mathcal{O})$ is the initial state (initial temperature distribution) in the region $\mathcal{O}$;

• •

$W$ is a cylindrical Wiener process in $L^{2}({\mathcal{O}})$;

• •

$\sigma\in{\mathcal{L}}(L^{2}({\mathcal{O}}))$.

Assume that this equation is well posed (in some suitable sense, see below for the precise setting) for every given ${\gamma_{0}(\cdot,\cdot)}$ in a suitable set of admissible control processes and denote its unique solution by ${y^{x,\gamma_{0}(\cdot,\cdot)}}$ to underline the dependence of the state $y$ on the control ${\gamma_{0}(\cdot,\cdot)}$ and on the initial datum $x$. The controller aims at minimizing, over the setof admissible controls, the objective functional

[TABLE]

where $\ell_{1},\ell_{2}:\mathbb{R}\rightarrow\mathbb{R}$ are given measurable functions bounded from below and $\lambda>0$ is a discount factor.

6.1.2 Infinite dimensional setting

We now rewrite the state equation (6.1) and the functional (6.2) in an infinite dimensional setting in the space $H\coloneqq L^{2}({\mathcal{O}})$. For more details, we refer to [16, Sec. 5] and references therein. Consider the realization of the Laplace operator with vanishing Neumann boundary conditions 151515To be precise, ${\mathcal{D}}(A_{N})$ is the closure in $H^{2}({\mathcal{O}})$ of the set of functions $\phi\in C^{2}(\overline{\mathcal{O}})$ having vanishing normal derivative at the boundary $\partial{\mathcal{O}}$.:

[TABLE]

It is well-known (see, e.g., [39, Ch. 3]) that $A_{N}$ generates a strongly continuous analytic semigroup \big{\{}e^{tA_{N}}\big{\}}_{t\geq 0} in $H$. Moreover, $A_{N}$ is a self-adjoint and dissipative operator. In particular $(0,+\infty)\subset\varrho(A_{N})$, where $\varrho(A_{N})$ denotes the resolvent set of $A_{N}$. So, if $\delta>0$, then $(\delta I-A_{N})$ is invertible and $(\delta I-A_{N})^{-1}\in\mathcal{L}(H)$. Moreover (see, e.g., [37, App. B]) the operator $(\delta I-A_{N})^{-1}$ is compact. Consequently, there exists an orthonormal complete sequence $\{e_{k}\}_{k\in\mathbb{N}}$ such that the operator $A_{N}$ is diagonal with respect to it:

[TABLE]

for a suitable sequence of eigenvalues $\{\mu_{k}\}_{k\in\mathbb{N}}\subseteq\mathbb{R}^{+}$ repeated according to their multiplicity (they are nonnegative due to dissipativity of $A_{N}$). We assume that such sequence is increasingly ordered. Then, $\mu_{0}=0$, as clearly the constant functions belong to Ker $(A_{N})$, and $\mu_{k}>0$ for each $k\in\mathbb{N}_{0}:=\mathbb{N}\setminus\{0\}$, since, as an immediate consequence of the Gauss-Green formula, only the constant functions belong to Ker $(A_{N})$. Moreover, [46, Sec. 5.6.2, p. 395] (see also [37, App. B]) provides also a growth rate for the sequence of eigenvalues; indeed

[TABLE]

We have (see, e.g., [37, App. B]) the isomorphic identification

[TABLE]

where $H^{s}({\mathcal{O}})$ denotes the Sobolev space of exponent $s\in\mathbb{R}$. Next, consider the following problem with Neumann boundary condition:

[TABLE]

Given any $\delta>0$ and $\alpha\in L^{2}(\partial{\mathcal{O}})$, there exists a unique solution $N_{\delta}\alpha\in H^{3/2}({\mathcal{O}})$ to (6.7). Moreover, the operator (Neumann map)

[TABLE]

is continuous (see [38, Th. 7.4]). So, in view of (6.6), the map

[TABLE]

is continuous. In [16, Sec. 5], it is shown that the natural abstract reformulation of the original control problem in the space $H$ is

[TABLE]

where $L_{N}^{\delta,\varepsilon}:=(\delta I-A_{N})^{\frac{3}{4}-\varepsilon}N_{\delta}\in\mathcal{L}(L^{2}(\partial{\mathcal{O}});H)$, $G_{N}^{\delta,\varepsilon}\coloneqq(\delta I-A_{N})^{\frac{1}{4}+\varepsilon}$, and $u(t)\coloneqq\gamma_{0}(t,\cdot)\in L^{2}(\partial{\mathcal{O}})$ for $t\geq 0$. We are now in the framework of (3.1), with $K=H$, $A=A_{N}$, $G=G_{N}^{\delta,\varepsilon}$, $L=L_{N}^{\delta,\varepsilon}$, and $U=L^{2}(\partial{\mathcal{O}})$. Let us consider, as set of admissible controls,

[TABLE]

where $\Lambda\subseteq L^{2}(\partial{\mathcal{O}})$ and $p$ will be specified later according to (3.4). Defining

[TABLE]

and

[TABLE]

the functional (6.2) can be rewritten in the Hilbert space framework as

[TABLE]

6.1.3 HJB equation and verification theorem

Setting $Q\coloneqq\sigma\sigma^{*}$, the HJB equation associated to the minimization of (6.11) is

[TABLE]

Since the semigroup \big{\{}e^{tA_{N}}\big{\}}_{t\geq 0} is strongly continuous and analytic, then by [42, Th. 6.13(c)] the operator $e^{tA_{N}}G_{N}^{\delta,\varepsilon}$ can be extended to $\overline{e^{tA_{N}}G_{N}^{\delta,\varepsilon}}=G_{N}^{\delta,\varepsilon}e^{tA_{N}}\in{\mathcal{L}}(H)$ for every $t>0$ and

[TABLE]

Hence, Assumption 3.1(i) and (iii) is satisfied with $A=A_{N}$, $G=G_{N}^{\delta,\varepsilon}$, and $\beta=\varepsilon+1/4$. Consequently, recalling (3.4), we choose $p>\frac{1}{\frac{3}{4}-\varepsilon}$.

Now, assume the following.

• (H1)

$\sigma$ satisfies Assumption 3.1(ii).

• (H2)

Conditions (A1) and (A2) of Remark 5.2 hold true with $G=G_{N}^{\delta,\varepsilon}$.

• (H3)

$\ell_{1}\in C_{b}(\mathbb{R})$, so $l_{1}\in C_{b}(H)$161616According to Remark 4.9 it is possible to deal with the case when $\ell_{1}$, and so $l_{1}$, has polynomial growth.. Moreover the map $q\mapsto F_{1}(q)$, defined by

[TABLE]

is Lipschitz continuous. These conditions imply that $F_{0}(x,q)=l_{1}(x)+F_{1}(q)$ satisfies condition (A3) of Remark 5.2.

Then, under such assumptions, by Remark 5.2, for sufficiently large $\lambda>0$ there exists a unique mild solution $v$ to (6.12). By definition of mild solution, we have $v\in C^{1,G}_{b}(H)$. Furthermore, Assumption A.2 is verified through Remark A.3 in this case. Hence Proposition A.4 applies yielding $v\in{\mathcal{S}}^{A,G}(H)$ and enabling the application of Theorem 5.6. We now discuss the validity of the above assumptions (H1)–(H3).

• •

On the validity of (H1). First of all, we note that in Assumption 3.1(ii), we can take $\gamma$ as small as we want; indeed, if this assumption holds true for some $\bar{\gamma}\in(0,1/2)$, then it holds true also for all $\gamma\in(0,\bar{\gamma})$. By (6.4), the operator $e^{tA_{N}}$ is diagonal with respect to the orthonormal basis $\{e_{k}\}$ with eigenvalues $e^{-t\mu_{k}}$. Assumption 3.1(ii) rewrites as

[TABLE]

Applying Fubini-Tonelli’s Theorem and considering (6.5) we see that (6.14) holds if

[TABLE]

Let $\theta\geq 0$ be such that

[TABLE]

(recall that $\sigma\in\mathcal{L}(H)$, so $\theta=0$ always verifies (6.16)). Considering that $\gamma$ can be taken as small as we want and combining (6.15) and (6.16), we conclude that (H1) holds if we may take in (6.16)

[TABLE]

In particular, if $d=1$, then (H1) holds true for all $\sigma\in\mathcal{L}(H)$.

• •

On the validity of (H2). By (6.5), we have, for $k\in\mathbb{N}$,

[TABLE]

The operator $\overline{e^{tA_{N}}G_{N}^{\delta,\varepsilon}}$ is diagonal too with respect to $\{e_{k}\}_{k\in\mathbb{N}}$ and

[TABLE]

Assume now further that $\sigma$ is diagonal with respect to $\{e_{k}\}_{k\in\mathbb{N}}$ and nondegenerate, i.e. $\sigma e_{k}=\sigma_{k}e_{k}$ for every $k\in\mathbb{N},$ where $\sigma_{k}>0$ for every $k\in\mathbb{N}$. Set $q_{k}\coloneqq\sigma_{k}^{2}>0$ for $k\in\mathbb{N}$. Then $Q_{t}$ is diagonal too. Moreover and $Q_{t}e_{0}=tq_{0}e_{0}$ and

[TABLE]

Hence, with the agreement $\frac{1-e^{-2\mu_{k}t}}{2\mu_{k}}\coloneqq t$ if $k=0$, we have

[TABLE]

Since |\Gamma_{G}(t)|_{\mathcal{L}(H)}=\sup_{k\in\mathbb{N}}\big{|}\Gamma_{G}(t)e_{k}\big{|}_{H}, then, with the agreement that $\frac{2\mu_{k}}{e^{2t\mu_{k}}-1}\coloneqq t^{-1}$ if $k=0$, conditions (A1) and (A2) of Remark 5.2 hold true if and only if

[TABLE]

Assume that

[TABLE]

and let $k_{0}\in\mathbb{N}$ and $c_{0}>0$ be such that $q_{k}\geq c_{0}k^{-2\theta}$ for some $c_{0}>0$ and every $k\geq k_{0}$. Considering (6.5), let $c_{1},c_{2}>0$ and $k_{0}^{\prime}\in\mathbb{N}$ be such that $c_{1}k^{\frac{2}{d}}\leq\mu_{k}\leq c_{2}k^{\frac{2}{d}}$ for every $k\geq k_{0}^{\prime}.$ Calling $\bar{k}\coloneqq k_{0}\vee k_{0}^{\prime}$ it is clear that, for a suitable $C_{0}>0$,

[TABLE]

Hence, to prove (6.19) above, we take $k\geq\bar{k}$ and we rewrite (6.19) (up to a constant depending on $c_{0},c_{1},c_{2}$) as

[TABLE]

Noting that ${C_{1}}\coloneqq\sup_{s>0}\frac{s^{\frac{3}{2}+2\varepsilon+d\theta}}{e^{s}-1}<+\infty$, we can estimate

[TABLE]

Therefore, (H2) is satisfied whenever (6.20) holds for some $\theta$ such that $\frac{3}{2}+2\varepsilon+d\theta<2$. As $\varepsilon>0$ can be taken arbitrarily small, we conclude that (H2) can be fulfilled if (6.20) holds for some $\theta$ such that

[TABLE]

• •

On the simultaneous validity of (H1)–(H2). Looking at (6.17) and (6.22), we see that (H1)-(H2) can be simultaneously fulfilled by choosing a suitable $\varepsilon>0$ if $\sigma$ is diagonal with respect to $\{e_{k}\}_{k\in\mathbb{N}}$ and (6.20) is verified for some $\theta\geq 0$ such that

[TABLE]

These requirements can be fulfilled only for dimension $d\leq 2$.

• •

On the validity of (H3). This is guaranteed, for instance, if $\Lambda$ is bounded, $\ell_{1}$ is continuous and bounded, $\ell_{2}$ is measurable.

6.1.4 Optimal Feedback Controls

In the framework of the previous subsection, we look now at the existence of optimal feedback controls.

Theorem 6.1**.**

Let (H1)–(H3) of the previous subsection hold. Assume that the multi-valued map

[TABLE]

admits a Lipschitz continuous selection $\psi$ and that $D^{G_{N}^{\delta,\varepsilon}}v$ is Lipschitz continuous. Set $\phi:=\psi\circ D^{G_{N}^{\delta,\varepsilon}}v$. Then the SDE

[TABLE]

admits a unique mild solution $X_{\phi}(\cdot;x)\in\mathcal{K}_{\mathcal{P}}^{1,loc}(H)$ (in the sense of (5.20)) admitting a version with continuous trajectories. As a consequence, Corollary 5.7(i) applies providing the optimality of the couple \big{(}{u_{\phi}}(\cdot),{X_{\phi}}(\cdot;x)\big{)}, where $u_{\phi}(t):=\phi(X_{\phi}(t;x))$ for $t\geq 0$.

Proof.

By the assumptions, the map $\phi$ is Lipschitz continuous too. Then the proof follows the classical fixed point arguments as in standard results of existence and uniqueness of SDEs in infinite dimension, see e.g. [10, Theorem 7.5]. Here we only need to take care of dealing with $\overline{e^{sA_{N}}G_{N}^{\delta,\varepsilon}}$ in place of $e^{sA_{N}}$ in the convolution term and use (3.2) with $G=G_{N}^{\delta,\varepsilon}$.  ${\square}$

The assumption that $\Psi$ defined in (6.24) admits a Lipschitz continuous selection $\psi$ is guaranteed, for example, if $\Lambda=U$, $l_{2}:U\rightarrow\mathbb{R}$ is strictly convex,

[TABLE]

$l_{2}$ is Fréchet differentiable, and $Dl_{2}$ has Lipschitz continuous inverse. Indeed, in this case the infimum in (6.24) is uniquely achieved (hence, $\Psi$ is single-valued) at

[TABLE]

Hence, if we are able to check that $D^{G_{N}^{\delta,\varepsilon}}v$ is Lipschitz continuous, we can then apply Corollary 5.7(i) in its strongest form to get uniqueness of the optimal control constructed.

On the other hand, checking that $D^{G_{N}^{\delta,\varepsilon}}v$ is Lipschitz continuous might be, in general, a very difficult task171717This can be done assuming more regularity of $\ell_{1}$ — hence of $l_{1}$ — and proving a suitable $C^{2}$ property of $v$. See, e.g., the approach used in [31] or in [29]., whereas mere continuity of $D^{G_{N}^{\delta,\varepsilon}}v$ is a condition already “contained” in the definition of mild solution to (6.12). Hence, it would be meaningful to provide a Peano type result 181818This is not straightforward: in infinite dimension Peano’s Theorem fails in general (see [24]). of existence of mild solutions to CLE (6.25). This seems possible when a selection $\psi$ of $\Psi$ in (6.24) is known to be only continuous and bounded on bounded sets, as

• (i)

the semigroup $\{{e^{tA_{N}}}\}_{t\geq 0}$ is compact;

• (ii)

as $D^{G_{N}^{\delta,\varepsilon}}v$ is continuous and bounded by construction, the map $\phi:=\psi\circ D^{G_{N}^{\delta,\varepsilon}}v$ is continuous and bounded.

Indeed, in such a framework, it seems possible to use the methods of [7, Prop. 3] (see also [22]), passing through the use of the so-called Skorohod representation theorem, to construct martingale solutions to (6.25); hence, to construct optimal feedback controls in the weak formulation.

Remark 6.2**.**

In the specific case we are handling, where the diffusion term is just additive in the equation, a way to construct the solution in the original probability space $\Omega$ might consist in constructing a pathwise solution dealing with a parameterized family of deterministic problems with parameter $\omega\in\Omega$ (see [2], [9, Sections 14.2 and 15.2], [19], [40]). Once this is done, the problem is to prove that the family of solutions constructed $\omega$ by $\omega$ admits an adapted selection. The existence of a selection measurable with respect to $\mathcal{F}$ can be obtained using measurable selection theorems (see again [2]); proving that this selection is also adapted is a problematic task, which is still open. In the case when one knows ex ante that the pathwise solution is unique for a.e. $\omega\in\Omega$, then F. Flandoli (personal communication) showed us how to accomplish this task. Unfortunately, in our case, the uniqueness of the solutions of the deterministic equations for a.e. $\omega\in\Omega$ only holds when the properties of the coefficients allow to find directly mild solutions to SDE (6.25).

6.2 Stochastic optimal control with delay in the control variable

Here we consider an infinite horizon version of a control problem studied in [29, 30]. Consider the following linear controlled one dimensional SDE:

[TABLE]

where

• •

$W=\{W(t)\}_{t\geq 0}$ is a standard one dimensional Brownian motion;

• •

$a_{0},b_{0},\sigma_{0}\in\mathbb{R}$, $\sigma_{0}>0$;

• •

$d>0$ represents the maximum delay the control takes to affect the system;

• •

$b_{1}(\cdot)$ is a (real-valued) function weighting the aftereffects of the control on the system; we consider here the case of distributed delay, i.e. when $b_{1}\in L^{2}([-d,0],\mathbb{R})$.

The initial data are the initial state $y_{0}$ and the past history $u_{0}$ of the control. The control $u$ takes values in a closed subset $\Lambda\subseteq U:=\mathbb{R}$ and belongs to $\mathcal{U}_{2}$ (defined by (3.5) with $p=2$).

Such kind of equations (even in a deterministic framework) have been used to model the effect of advertising on the sales of a product [27, 28, 17], the effect of investments with time to build on growth [13, 1], to model optimal portfolio problems with execution delay [3], to model the interaction of drugs with tumor cells [35, p. 17].

Denoting by $y^{y_{0},u_{0},u(\cdot)}$ the unique solution to (6.26), the goal of the problem is to minimize, over all control strategies in $\mathcal{U}_{2}$, the following objective functional

[TABLE]

where $\ell_{0}:\mathbb{R}\rightarrow\mathbb{R}$ and $\ell_{1}:\Lambda\rightarrow\mathbb{R}$ are measurable and bounded from below. It is important to note that here $\ell_{0}$ and $\ell_{1}$ do not depend on the past of the state and/or control. This is a very common feature of many applied problems.

A standard way to approach these delayed control problems, introduced in [47] for the deterministic case and extended to the stochastic case in [27], is to reformulate them as equivalent infinite dimensional control problems without delay191919It must be noted that, under suitable restrictions on the data, one can treat (stochastic) optimal control problems with delay avoiding to look at them as infinite dimensional systems (see [18]). However, this is possible only in very special cases, leaving out a lot of of concrete applications.. The details are given in [29] for the finite horizon case, which is completely similar to the infinite horizon case, with the obvious changes (see also [17] for the infinite horizon case in a deterministic framework with a different embedding space). Consider the Hilbert space $H:=\mathbb{R}\times L^{2}([-d,0],\mathbb{R})$, set $b:=(b_{0},b_{1}(\cdot))\in H$, and assume, without loss of generality, $|b|_{H}=1$. The state equation (6.26) is rephrased in $H$ as a linear SDE with state variable $X=(X_{0},X_{1}(\cdot))$ as follows:

[TABLE]

where

[TABLE]

[TABLE]

and the initial datum ${x}$ is defined as

[TABLE]

It is well known that $A$ is the generator of a $C_{0}$-semigroup of linear bounded operators on $H$. Note that the infinite dimensional datum $x_{1}(\cdot)$ depends on the “initial past” $u_{0}(\cdot)$ of the control. It turns out that $X_{0}(t;x,u(\cdot))=y^{y_{0},u_{0},u(\cdot)}$, so (6.27) is rewritten as

[TABLE]

Setting $Q:=\sigma\sigma^{*}$, the HJB equation associated to the minimization of (6.29) is

[TABLE]

Notice that $D^{G}=\frac{\partial}{\partial{b}}$, where the latter symbol denotes the directional derivative along the direction $b$. So, the nice feature of the equation above is that the nonlinearity on the gradient only involves the directional derivative $D^{G}$. Note also that here we do not have the so called structural condition $G(\mathbb{R})\subseteq\sigma(\mathbb{R})$; this prevents the use of techniques based on Backward SDEs (see, e.g., [21]) to tackle the problem.

Now we check if the assumptions of our main result Theorem 5.6 are verified. First of all, it is easy to check that Assumption 3.1 and Assumption A.5 hold. The third assumption, i.e. the existence of a mild solution $v\in\mathcal{S}^{A,G}(H)$ to (6.30) needs to be discussed.

In [29], the authors study a finite horizon optimal control problem with the same state equation (6.26) and a similar objective functional. Exploiting only partial smoothing properties of the transition semigroup associated to the state equation (6.28) with null control, the authors are able to provide, under suitable reasonable assumptions on the data, existence and uniqueness results for the parabolic HJB equation associated to the control problem.

We believe that the approach of [29] can be adapted to our infinite horizon case, getting a mild solution $v\in\mathcal{D}(\mathcal{A}^{(0)})\cap C^{1,G}_{b}(H)$ to HJB (6.30). Then, to apply our theory one should prove that such function $v$ is Lipschitz continuous on compact sets, which enables to apply Proposition A.6 to get $v\in\mathcal{S}^{A,G}(H)$. To get this goal one can proceed as in [29] by assuming more regularity on the data of the problem. More precisely, assuming that $l_{0}\in C^{1}_{b}(\mathbb{R})$ and that the Hamiltonian $p\mapsto\inf_{u\in\Lambda}\left\{up+\ell_{1}(u)\right\}$ is differentiable with Lipschitz continuous derivative, [29] proves that the mild solution $v\in C^{1}_{b}(H)$. This fact, in particular, implies the required Lipschitz continuity of $v$. In [30] the authors also provide a verification theorem for their finite horizon problem. They use an approximation procedure of the solution of the HJB equation, which our results allow to avoid here.

Appendix A Appendix

Recall that, given $G\in{\mathcal{L}}_{u}(K,H)$, the pseudo-inverse $G^{-1}:{\mathcal{R}}(G)\rightarrow{\mathcal{D}}(G)$ is defined as the operator that associates to each $h\in{\mathcal{R}}(G)$ the element of $G^{-1}(\{h\})$ having minimum norm.202020Existence and uniqueness of such an element follows from the fact that $G$ is a closed operator and applying the results of [11, Sec. II.4.29, p. 74]). Note that $G^{-1}G:{\mathcal{D}}(G)\rightarrow{\mathcal{D}}(G)$ is bounded, so it can be extended to a bounded operator $\overline{G^{-1}G}\in{\mathcal{L}}(K)$.

Lemma A.1**.**

We have

[TABLE]

Proof.

Assume first that $k\in{\mathcal{D}}(G)$. In this case $GG^{-1}Gk=Gk$. Then, using Remark 2.4, we write

[TABLE]

If $k\notin{\mathcal{D}}(G)$, we can take a sequence $\{k_{n}\}\subseteq{\mathcal{D}}(G)$ converging to $k$. Considering (A.1) on $k_{n}$ and passing to the limit the claim follows taking into account that $\overline{G^{-1}G}$ is bounded.  ${\square}$

Assumption A.2**.**

The operator $G\in{\mathcal{L}}_{u}(K,H)$ is such that for every $k\in K$

• (i)

*there exists $\varepsilon>0$ such that $\left\{\displaystyle\int_{0}^{t}\overline{e^{sA}G}k\,ds\right\}_{t\in(0,\varepsilon)}\subseteq{\mathcal{R}}(G)$; *

• (ii)

$G^{-1}\left(\frac{1}{t}\displaystyle\int_{0}^{t}\overline{e^{sA}G}k\,ds\right)\rightarrow\overline{G^{-1}G}k$, as $t\rightarrow 0^{+}$.

Remark A.3**.**

Note that $\displaystyle\int_{0}^{t}e^{sA}h\,ds\in{\mathcal{D}}(A)$ for every $t>0$ and $h\in H$. So, in view of the fact that $\overline{G^{-1}G}$ is bounded, Assumption A.2 is verified, in particular, if $K=H$, ${\mathcal{D}}(A)\subseteq{\mathcal{D}}(G)$ and, for sufficiently small $\varepsilon>0$,

[TABLE]

This applies, e.g., to the case when $A$ is dissipative and generates an analytic semigroup, and $G=(\delta I-A)^{\beta}$ with $\delta>0$ and $\beta\in(0,1)$ (see the example of Section 6.1).

Proposition A.4**.**

Let Assumption A.2 holds. Then $\mathcal{S}^{A,G}(H)=C^{1,G}_{b}(H)$.

Proof.

Fix $k\in K$, $z\in C(\mathbb{R}^{+};H)$ and let $\varepsilon>0$ be as in Assumption A.2(i). Noting that $GG^{-1}h=h$ for every $h\in{\mathcal{R}}(G)$, by Assumption A.2(i) we can write

[TABLE]

Moreover, by Assumption A.2(ii), we have

[TABLE]

Fix now $t\in(0,\varepsilon)$. Using (A.2) we write

[TABLE]

Mean value theorem applied to the function $[0,1]\rightarrow\mathbb{R},\ \xi\mapsto f\left(x(t)+\xi Gk{(t)}\right)$ yields (see also Remark 2.4)

[TABLE]

Hence, (A) rewrites as

[TABLE]

Moreover, we can estimate

[TABLE]

Now we are going to take the limit for $t\rightarrow 0^{+}$ in (A). To this purpose, we observe that, as $D^{G}\varphi\in C_{b}(H,K)$ and \big{\{}z(t)\big{\}}_{t\in(0,\varepsilon)} is compact in $H$, we have

[TABLE]

By definition of $k(t)$ (see (A.2)), we have $|Gk(t)|_{H}\stackrel{{\scriptstyle t\rightarrow 0^{+}}}{{\longrightarrow}}0$. Hence, (A.7) provides

[TABLE]

Hence, combining (A.3), (A.6) and (A.8), we get

[TABLE]

Moreover, (A.3) and the continuity of the maps $t\mapsto z(t)$ and $x\mapsto D^{G}\varphi(x)$ entails

[TABLE]

Combining (A), (A.9), (A.10), and Lemma A.1, the claim follows.  ${\square}$

Assumption A.5**.**

$G\in{\mathcal{L}}(K,H)$.

Proposition A.6**.**

Let Assumption A.5 hold and let $\varphi\in C^{1,G}_{b}(H)$ be Lipschitz continuous on compact sets. Then $\varphi\in\mathcal{S}^{A,G}(H)$.

Proof.

Let $k\in K$. Observe that, as $G\in{\mathcal{L}}(K,H)$, we have $k\in K={\mathcal{D}}(G)$, $\overline{e^{sA}G}k=e^{sA}Gk$ for every $s>0$, and

[TABLE]

Let $t>0$. We can split

[TABLE]

The set \bigg{\{}z(t)+\displaystyle\int_{0}^{t}{e^{sA}G}k\,ds\bigg{\}}_{t\in(0,1)}\bigcup\,\bigg{\{}z(t)+tGk\bigg{\}}_{t\in(0,1)}\subset K is precompact. Hence, by Lipschitz continuity of $\varphi$ on compact sets, we have for some $C_{0}>0$ independent of $t\in(0,1)$

[TABLE]

We let now $t\rightarrow 0^{+}$ in (A). Combining with (A.13) and (A.11) we get

[TABLE]

provided that the limit in the right hand side above exists, as we are going to show. We write

[TABLE]

By the equalities above and considering that $D^{G}\varphi\in C_{b}(H;K)$, we have

[TABLE]

and the claim follows from (A.14).

${\square}$