Ergodic control of diffusions with compound Poisson jumps under a general structural hypothesis

TL;DR

This paper extends ergodic control theory for jump diffusions driven by compound Poisson processes, providing a full characterization of optimality, regularity of solutions, and practical methods for near-optimal control in complex network applications.

Contribution

It generalizes previous results to non-near-monotone costs, characterizes optimal controls via HJB equations, and offers practical approaches for scheduling in large-scale networks.

Findings

Full characterization of optimality via HJB equations.

Regularity results for solutions under mild conditions.

Method to construct near-optimal controls by solving HJB on large domains.

Abstract

We study the ergodic control problem for a class of controlled jump diffusions driven by a compound Poisson process. This extends the results of [SIAM J. Control Optim. 57 (2019), no. 2, 1516-1540] to running costs that are not near-monotone. This generality is needed in applications such as optimal scheduling of large-scale parallel server networks. We provide a full characterization of optimality via the Hamilton-Jacobi-Bellman (HJB) equation, for which we additionally exhibit regularity of solutions under mild hypotheses. In addition, we show that optimal stationary Markov controls are a.s. pathwise optimal. Lastly, we show that one can fix a stable control outside a compact set and obtain near-optimal solutions by solving the HJB on a sufficiently large bounded domain. This is useful for constructing asymptotically optimal scheduling policies for multiclass parallel server networks.

Equations

$$\operatorname*{osc}_{D}\,f\,:=\,\sup\,\bigl{\{}f(x)-f(y)\,\colon\,x,\,y\in D\bigr{\}}\,.$$

$$\text{uptau}(A):=\inf \{t>0:X_t\notin A\}.$$

$$\mathrm{d}{X}_t:=b(X_t,U_t)\mathrm{d}t+\text{upsigma}(X_t)\mathrm{d}{W}_t+\mathrm{d}{L}_t$$

$$\mathrm{d}{L}_t:={\int }_{\text{mathds}{R}_{\ast }^l}g(z)\tilde{\mathcal{N}}(\mathrm{d}t,\mathrm{d}z)$$

$${\mathfrak{F}}_s:=\text{the completion of }\sigma \{X_0,U_r,W_r,\mathcal{N}(r,\cdot ):r\le s\}\text{ relative to }(\mathfrak{F},P).$$

$$\mathrm{d}{X}_t^{\circ }:=b(X_t^{\circ },v(t,X_t^{\circ }))\mathrm{d}t+\text{upsigma}(X_t^{\circ })\mathrm{d}{W}_t,$$

$${\mathcal{A}}\varphi(x,u)\,:=\,a^{ij}(x)\partial_{ij}\varphi(x)+\widetilde{b}^{i}(x,u)\partial_{i}\varphi(x)+\int_{\mathds{R}^{d}}\bigl{(}\varphi(x+y)-\varphi(x)\bigr{)}\,\nu(\mathrm{d}y)\,,$$

$${\tilde{\mathcal{L}}}_u\phi (x):=a^{ij}(x){\partial }_{ij}\phi (x)+{\tilde{b}}^i(x,u){\partial }_i\phi (x)-\mathbf{\nu }\phi (x),\text{and}\tilde{\mathcal{I}}\phi (x):={\int }_{\text{mathds}{R}^d}\phi (x+y)\nu (\mathrm{d}y).$$

$$\operatorname{\mathbb{E}}_{x}^{U}[f(X_{t\wedge\uptau(D)})]\,=\,f(x)+\operatorname{\mathbb{E}}_{x}^{U}\biggl{[}\int_{0}^{t\wedge\uptau(D)}{\mathcal{A}}f(X_{s},U_{s})\,\mathrm{d}{s}\biggr{]}$$

$$\varrho^{\vphantom{\frac{1}{2}}}_{U}(x)\,:=\,\limsup_{T\rightarrow\infty}\,\frac{1}{T}\,\operatorname{\mathbb{E}}_{x}^{U}\biggl{[}\int_{0}^{T}{\mathscr{R}}(X_{t},U_{t})\,\mathrm{d}{t}\biggr{]}\,.$$

$${\varrho }_{\ast }(x):=\underset{U\in \mathfrak{U}}{\inf }{\varrho }_U^{}(x).$$

$$\begin{array}{rl}{\mathcal{A}}_u{\mathcal{V}}_{\circ }(x)& \le \text{mathds}{1}_{{\mathcal{B}}_{\circ }}(x)-F(x,u)\forall (x,u)\in {\mathcal{K}}^c\times \mathbb{U},\\ {\mathcal{A}}_u{\mathcal{V}}_{\circ }(x)& \le \text{mathds}{1}_{{\mathcal{B}}_{\circ }}(x)+\mathcal{R}(x,u)\forall (x,u)\in \mathcal{K}\times \mathbb{U}.\end{array}$$

$${\mathcal{A}}_{\hat{v}}\mathcal{V}(x)\le \hat{\kappa }\text{mathds}{1}_{{\mathcal{B}}_{\circ }}(x)-{\mathcal{R}}_{\hat{v}}(x),\forall x\in \text{mathds}{R}^d,$$

$${\varrho }_U^{}(x)<\infty \text{for some }x\in \text{mathds}{R}^d\text{ and }U\in \mathfrak{U},$$

$${\mathbb{U}}\,:=\,\bigl{\{}u=(u^{c},u^{s})\in\mathds{R}^{d}_{+}\times\mathds{R}^{J}_{+}\colon\langle e,u^{c}\rangle=\langle e,u^{s}\rangle=1\bigr{\}}\,.$$

$$\mathrm{d}{X}_t=b(X_t,U_t)\mathrm{d}t+\sigma \mathrm{d}{W}_t+\theta \mathrm{d}{L}_t,$$

$$b(x,u)\,=\,\ell-M_{1}\bigl{(}x-\langle e,x\rangle^{+}u^{c}\bigr{)}-\langle e,x\rangle^{+}\varGamma u^{c}+\langle e,x\rangle^{-}M_{2}u^{s}$$

$$\mathcal{R}(x,u):=\sum _{i\in \mathcal{I}}{c}_i[⟨e,x{⟩}^{+}{u}_i^c{]}^m+\sum _{j\in \mathcal{J}}{s}_j[⟨e,x{⟩}^{-}{u}_j^s{]}^m$$

$$\bigl{\langle}\widetilde{b}(x,u),\nabla\mathscr{V}_{Q,m}(x)\bigr{\rangle}\,\leq\,\begin{cases}C_{1}-m\langle x,Qx\rangle^{\nicefrac{{m}}{{2}}-1}\lvert x\rvert^{2}&\forall\,(x,u)\in{\mathcal{K}}_{\delta}^{c}\times{\mathbb{U}}\,,\\[5.0pt] C_{1}\bigl{(}1+\lvert\langle e,x\rangle\rvert^{m}\bigr{)}&\forall\,(x,u)\in{\mathcal{K}}_{\delta}\times{\mathbb{U}}\end{cases}$$

$$\displaystyle\int_{\mathds{R}^{d}}\bigl{(}\mathscr{V}_{Q,m}(x+z)-\mathscr{V}_{Q,m}(x)\bigl{)}\,\nu(\mathrm{d}{z})$$

$$\le C_2+ϵ⟨x,Qx{⟩}^{\text{nicefrac}m2}$$

$$M_1=\left[\begin{array}{ccc}{\mu }_{11}& 0& 0\\ {\mu }_{22}-{\mu }_{21}& {\mu }_{22}& 0\\ 0& 0& {\mu }_{32}\end{array}\right],M_2=\left[\begin{array}{cc}0& 0\\ {\mu }_{21}-{\mu }_{22}& 0\\ 0& 0\end{array}\right]$$

$$b(x,u)\,=\,\ell-M\bigl{(}x-\langle e,x\rangle^{+}u\bigr{)}-\langle e,x\rangle^{+}\varGamma u\,,$$

$${\int }_{\text{mathds}{R}^d}{\mathcal{A}}_{u}f(x)\text{uppi}(\mathrm{d}x,\mathrm{d}u)=0\forall f\in \mathcal{C}.$$

$$\bar{\zeta}^{U}_{x,t}(f)\,=\,\int_{\mathds{R}^{d}\times{\mathbb{U}}}f(x,u)\,\bar{\zeta}^{U}_{x,t}(\mathrm{d}{x},\mathrm{d}{u})\,:=\,\frac{1}{t}\int_{0}^{t}\operatorname{\mathbb{E}}_{x}^{U}\biggl{[}\int_{{\mathbb{U}}}f(X_{s},u)\,U_{s}(\mathrm{d}{u})\biggr{]}\,\mathrm{d}{s}\,,$$

$${\zeta }_t^U(f)={\int }_{\text{mathds}{R}^d\times \mathbb{U}}f(x,u)\mathrm{d}{\zeta }_t^U(\mathrm{d}x,\mathrm{d}u):=\frac{1}{t}{\int }_0^t{\int }_{\mathbb{U}}f(X_s,u)U_s(\mathrm{d}u)\mathrm{d}s,$$

$$\widetilde{{\mathcal{K}}}\,:=\,({\mathcal{K}}\times{\mathbb{U}})\cup\bigl{\{}(x,u)\in\mathds{R}^{d}\times{\mathbb{U}}\,\colon{\mathscr{R}}(x,u)>F(x,u)\bigr{\}}\,.$$

$${\mathcal{A}}_u{\mathcal{V}}_{\circ }(x,u)\le \text{mathds}{1}_{{\mathcal{B}}_{\circ }}(x,u)-F(x,u)\text{mathds}{1}_{{\tilde{\mathcal{K}}}^c}(x,u)+\mathcal{R}(x,u)\text{mathds}{1}_{\tilde{\mathcal{K}}}(x,u),\forall (x,u)\in \text{mathds}{R}^d\times \mathbb{U}.$$

$${\mathscr{R}}\,\leq\,\widetilde{F}\,\leq\,\tilde{\kappa}\,\bigl{(}\mathds{1}_{{\mathscr{B}}_{\circ}}+{\mathscr{R}}\,\mathds{1}_{\widetilde{{\mathcal{K}}}}+F\,\mathds{1}_{{\widetilde{{\mathcal{K}}}^{c}}}\bigr{)}$$

$$\bar{\zeta}^{U}_{x,t}({\mathscr{R}}^{\epsilon})\,\leq\,\bar{\zeta}^{U}_{x,t}({\mathscr{R}})+\epsilon\tilde{\kappa}\biggl{(}1+\frac{1}{t}\mathscr{V}_{\!\circ}(x)+2\bar{\zeta}^{U}_{x,t}({\mathscr{R}})\biggr{)}\qquad\forall\,(x,t)\in{\mathds{R}^{d}}\times(0,\infty)\,,\ \forall\,U\in{\mathfrak{U}}\,.$$

$${\text{uppi}}_v({\mathcal{R}}^{ϵ})\le {\varrho }_v+ϵ\tilde{\kappa }(1+2{\varrho }_v)$$

Full text

We provide a full characterizations of optimality via the Hamilton–Jacobi–Bellman (HJB) equation, for which we additionally exhibit regularity of solutions under mild hypotheses. In addition, we show that optimal stationary Markov controls are a.s. pathwise optimal. Lastly, we show that one can fix a stable control outside a compact set and obtain near-optimal solutions by solving the HJB on a sufficiently large bounded domain. This is useful for constructing asymptotically optimal scheduling policies for multiclass parallel server networks.

1. Introduction

Control problems for jump diffusions have been studied extensively. We refer the readers to [BenLi-84] and references therein for the study of the discounted problem and many applications. In [Menaldi-99], the ergodic control problem under a strong blanket stability condition (see [Menaldi-99, (1.6)]) has been studied. In [ACPZ19], the authors have studied the ergodic control problem for jump diffusions when the associated Lévy measures are finite and state-dependent and have rough kernels under a near-monotone running cost function. However, in many applications the dynamics are not stable under any Markov control, nor do they have a near-monotone running cost function. In this paper we waive these assumptions, and study the ergodic control problem under the more general structural hypotheses (see Assumptions 2.1 and 2.2) first introduced in [ABP15], and also used in [AP16a] in the study of multiclass multi-pool queueing networks.

The class of jump diffusions studied in this paper is abstracted from the diffusion limit of multiclass queueing networks in the Halfin–Whitt regime with service interruptions [APS19]. The jump process in this model is compound Poisson, and thus the associated Lévy measure is finite. However, it not have any particular regularity properties such as density. In addition, the running cost function, which typically penalizes the queue size, is not near-monotone. We abstract and generalize this model, and consider a large class of diffusions with jumps, which includes models having a near-monotone running cost function, or with uniformly stable dynamics as special cases.

We first establish the existence of an optimal stationary Markov control for the ergodic control problem, and characterize all optimal stationary Markov controls via the ergodic Hamilton–Jacobi–Bellman (HJB) equation.

It is shown in [ACPZ19, Example 1.1] that the Harnack property may fail for infinitesimal generators of jump diffusions with compound Poisson jumps. Thus the approach developed in [ABG12, ABP15] for the study of the ergodic HJB equation associated with continuous diffusions cannot be applied here. On the other hand, the running cost function is assumed near-monotone in [ACPZ19], and thus the infimum of the value function for the discounted problem is attained in a compact set (see [ACPZ19, Theorem 3.2]), and the solutions of the ergodic HJB equation are bounded from below. In the present paper, we extend the technique developed in [ACPZ19], and derive the ergodic HJB under Assumptions 2.1 and 2.2. This is rather delicate, and requires an estimate of the negative part of the solutions of the HJB.

Another difficulty concerns the regularity of solutions of the discounted and ergodic HJB equations associated with jump diffusions, when the Lévy kernel is rough. In [ACPZ19], we show that the solutions have locally Hölder continuous second order derivatives when the Lévy measure has a compact support (see [ACPZ19, Remark 3.4]). In this paper, we present a gradient estimate for solutions of a class of second order nonlocal equations in Lemma 5.3 using scaling, and employ this to establish $\mathcal{C}^{2,\alpha}$ regularity of the solutions of the HJB equations in Theorem 5.3.

We also study pathwise optimality of optimal controls for the ergodic control problem. For continuous diffusion processes, pathwise optimality has been studied in [PMB00, PWM04, ABG12, AA17, Ghosh-97]. Pathwise optimality for jump diffusions with near-monotone running cost is studied in [ACPZ19, Theorem 4.4]. We extend the technique in [AA17], using also the result on convergence of random empirical measures for jump diffusions in [ACPZ19, Lemma 4.3] while providing a crucial estimate on the nonlocal term, to establish pathwise optimality for the model studied in this paper.

The ability to synthesize a near-optimal Markov control, by fixing a suitable stable control outside a large ball and solving the HJB equation inside the ball plays a crucial role in the study of asymptotic optimality for multiclass parallel server networks. This is used in [ABP15, AP18, AP19, ADPZ19] to construct asymptotically near-optimal scheduling policies for the prelimit system. In addressing this problem for jump diffusions, we first derive a lower bound for supersolutions of a general class of integro-differential equations in Lemma 7.1, and then use this to establish the required result in Theorems 7.1 and 7.1. In turn, this result is used to establish the asymptotic optimality of multiclass networks with service interruptions in [APZ19b].

1.1. Organization of the paper

In the next subsection, we summarize the notation used in this paper. In Section 2, we introduce the model and state the assumptions. Section 3 contains some examples from queueing networks whose limiting controlled jump diffusions satisfy these assumptions. Section 4 concerns the existence of optimal stationary Markov controls. Section 5 is devoted to the study of the HJB equations on the discounted and ergodic control problems. In Section 6, we study the pathwise optimality for the ergodic control problem. The characterization of near-optimal controls is studied in Section 7.

1.2. Notation

The standard Euclidean norm in $\mathds{R}^{d}$ is denoted by $\lvert\,\cdot\,\rvert$, $\langle\,\cdot\,,\cdot\,\rangle$ denotes the inner product, and $x^{\mathsf{T}}$ denotes the transpose of $x\in{\mathds{R}^{d}}$. The set of nonnegative real numbers is denoted by $\mathds{R}_{+}$, $\mathds{N}$ stands for the set of natural numbers, and $\mathds{1}$ denotes the indicator function. The minimum (maximum) of two real numbers $a$ and $b$ is denoted by $a\wedge b$ ($a\vee b$), respectively, and $a^{\pm}:=(\pm a)\vee 0$. The closure, boundary, and the complement of a set $A\subset{\mathds{R}^{d}}$ are denoted by $\bar{A}$, $\partial{A}$, and $A^{c}$, respectively. We also let $e:=(1,\dotsc,1)^{\mathsf{T}}$. For any function $f\colon\mathds{R}^{d}\to\mathds{R}$ and domain $D\subset\mathds{R}$ we define the oscillation of $f$ on $D$ as follows:

[TABLE]

We denote by $\uptau(A)$ the first exit time of the process $\{X_{t}\}$ from the set $A\subset\mathds{R}^{d}$, defined by

[TABLE]

The open ball of radius $r$ in $\mathds{R}^{d}$, centered at $x\in\mathds{R}^{d}$ is denoted by $B_{r}(x)$. We write $B_{r}$ for $B_{r}(0)$, and let $\uptau_{r}:=\uptau(B_{r})$, and ${\breve{\uptau}}_{r}:=\uptau(B^{c}_{r})$.

The term domain in $\mathds{R}^{d}$ refers to a nonempty, connected open subset of the Euclidean space $\mathds{R}^{d}$. For a domain $D\subset\mathds{R}^{d}$, the space $\mathcal{C}^{k}(D)$ ($\mathcal{C}^{\infty}(D)$), $k\geq 0$, refers to the class of all real-valued functions on $D$ whose partial derivatives up to order $k$ (of any order) exist and are continuous. By $\mathcal{C}^{k,\alpha}(\mathds{R}^{d})$ we denote the set of functions that are $k$-times continuously differentiable and whose $k$-th derivatives are locally Hölder continuous with exponent $\alpha$. The space ${L}^{p}(D)$, $p\in[1,\infty)$, stands for the Banach space of (equivalence classes of) measurable functions $f$ satisfying $\int_{D}\lvert f(x)\rvert^{p}\,\mathrm{d}{x}<\infty$, and ${L}^{\infty}(D)$ is the Banach space of functions that are essentially bounded in $D$. The standard Sobolev space of functions on $D$ whose generalized derivatives up to order $k$ are in ${L}^{p}(D)$, equipped with its natural norm, is denoted by ${\mathscr{W}}^{k,p}(D)$, $k\geq 0$, $p\geq 1$. In general, if $\mathcal{X}$ is a space of real-valued functions on $Q$, $\mathcal{X}_{\mathrm{loc}}$ consists of all functions $f$ such that $f\varphi\in\mathcal{X}$ for every $\varphi\in\mathcal{C}_{\mathrm{c}}^{\infty}(Q)$. In this manner we obtain for example the space ${\mathscr{W}}_{\text{loc}}^{2,p}(Q)$.

For $k\in\mathds{N}$, we let $\mathds{D}^{k}:=\mathds{D}(\mathds{R}_{+},\mathds{R}^{k})$ denote the space of $\mathds{R}^{k}$-valued cádlág functions on $\mathds{R}_{+}$. When $k=1$, we write $\mathds{D}$ for $\mathds{D}^{k}$.

For a nonnegative function $g\in\mathcal{C}(\mathds{R}^{d})$ we let ${\mathscr{O}}(g)$ denote the space of functions $f\in\mathcal{C}(\mathds{R}^{d})$ satisfying $\sup_{x\in\mathds{R}^{d}}\,\frac{\lvert f(x)\rvert}{1+g(x)}<\infty$. We also let ${\mathfrak{o}}(g)$ denote the subspace of ${\mathscr{O}}(g)$ consisting of those functions $f$ satisfying $\limsup_{\lvert x\rvert\to\infty}\,\frac{\lvert f(x)\rvert}{1+g(x)}=0$.

For a probability measure $\mu$ in ${\mathcal{P}}(\mathds{R}^{d})$, the space of Borel probability measures on $\mathds{R}^{d}$ under the Prokhorov topology, and a real-valued function $f$ which is integrable with respect to $\mu$ we use the notation $\mu(f):=\int_{{\mathds{R}^{d}}}f(x)\,\mu(\mathrm{d}{x})$.

2. The model and assumptions

We consider a controlled jump diffusion process ${\{X_{t}\}_{t\geq 0}}$ taking values in the $d$-dimensional Euclidean space $\mathds{R}^{d}$ defined by

[TABLE]

with $X_{0}=x\in\mathds{R}^{d}$. All random processes in Eq. 2.1 are defined on a complete probability space $(\Omega,\mathfrak{F},\operatorname{\mathbb{P}})$. The process ${\{W_{t}\}_{t\geq 0}}$ is a $d$-dimensional standard Wiener process, and ${\{L_{t}\}_{t\geq 0}}$ is a Lévy process defined as follows. Let $\widetilde{\mathcal{N}}(\mathrm{d}{t},\mathrm{d}{z})$ denote a martingale measure on $\mathds{R}^{l}_{*}=\mathds{R}^{l}\setminus\{0\}$, $l\geq 1$, taking the form $\widetilde{\mathcal{N}}(\mathrm{d}{t},\mathrm{d}{z})={\mathcal{N}}(\mathrm{d}{t},\mathrm{d}{z})-\Pi(\mathrm{d}{z})\mathrm{d}{t}$, where ${\mathcal{N}}$ is a Poisson random measure, and $\Pi(\mathrm{d}{z})\mathrm{d}{t}$ is the corresponding intensity measure, with $\Pi$ a finite measure on $\mathds{R}^{l}_{*}$. Then, ${\{L_{t}\}_{t\geq 0}}$ is given by

[TABLE]

for a measurable function $g\colon\mathds{R}^{d}\times\mathds{R}^{l}\to\mathds{R}^{d}$. The control process ${\{U_{t}\}_{t\geq 0}}$ takes values in a compact, metrizable space ${\mathbb{U}}$, $U_{t}(\omega)$ is jointly measurable in $(t,\omega)\in[0,\infty)\times\Omega$, and is non-anticipative: for $s<t$, \bigl{(}W_{t}-W_{s},\,{\mathcal{N}}(t,\cdot)-{\mathcal{N}}(s,\cdot)\bigr{)} is independent of

[TABLE]

Such a process $U$ is called an admissible control, and we let ${\mathfrak{U}}$ denote the set of admissible controls. We also assume that the initial conditions $X_{0}$, $W_{0}$ and ${\mathcal{N}}(0,\cdot)$ are independent.

To guarantee the existence of a solution to the equation Eq. 2.1, we impose the following usual assumptions on the drift, matrix $\upsigma$ and jump functions (compare with [ACPZ19]*Section 4.2). The functions $b\colon\mathds{R}^{d}\times{\mathbb{U}}\mapsto\mathds{R}^{d}$ and $\upsigma=[\upsigma^{ij}]\colon\mathds{R}^{d}\mapsto\mathds{R}^{d\times d}$ are continuous and have at most affine growth on $\mathds{R}^{d}$. Also, $b$ is locally Lipschitz continuous in its first argument uniformly with respect to the second. The matrix $\upsigma$ is locally Lipschitz continuous and nonsingular. We also assume that $\int_{\mathds{R}^{l}_{*}}\lvert g(z)\rvert^{2}\,\Pi(\mathrm{d}{z})<\infty$. Define \nu(A):=\Pi\bigl{(}\{z\in\mathds{R}_{*}^{l}\,\colon g(z)\in A\}\bigr{)}. Thus, $\nu$ is a Radon measure on $\mathds{R}^{d}$, and we let $\bm{\nu}:=\nu(\mathds{R}^{d})=\Pi(\mathds{R}^{l}_{*})$, which is finite. These hypotheses are enforced throughout the rest of the paper.

Under the above assumptions on the parameters, Eq. 2.1 has a unique strong solution under any admissible control $U$ (see, e.g., [GS72]*Part II, § 7), which is right continuous w.p.1, and has the strong Feller property. Recall that Markov controls may be identified with Borel measurable map $v$ on $\mathds{R}_{+}\times\mathds{R}^{d}$, by letting $U_{t}=v(t,X_{t})$. For any such Markov control $v$, define the associated diffusion process $\{X^{\circ},t\geq 0\}$ by

[TABLE]

with $X^{\circ}_{0}=x^{\circ}\in\mathds{R}^{d}$. It is well known that Eq. 2.2 has a pathwise unique strong solution [Gyongy-96]*Theorem 2.4. Since $\Pi$ is finite, it follows by the construction of a solution in [Skorokhod-89]*Chap. 1, Theorem 14 via Eq. 2.2 that Eq. 2.1 has a unique strong solution under any Markov control. We say that a Markov control $v$ is stationary if $v(t,x)$ is independent of $t$, and we use the symbol ${{\mathfrak{U}}_{\mathrm{sm}}}$ to denote the set of these controls.

For $\varphi\in\mathcal{C}^{2}(\mathds{R}^{d})$, define the integro-differential operator ${\mathcal{A}}\colon\mathcal{C}^{2}(\mathds{R}^{d})\to\mathcal{C}(\mathds{R}^{d}\times{\mathbb{U}})$ by

[TABLE]

where $a:=\frac{1}{2}\upsigma\upsigma^{\mathsf{T}}$, and $\widetilde{b}(x,u):=b(x,u)+\int_{\mathds{R}^{d}}z\,\nu(\mathrm{d}z)$. With $u\in{\mathbb{U}}$ treated as a parameter, we also define ${\mathcal{A}}_{u}\varphi(x):={\mathcal{A}}\varphi(x,u)$. We decompose this operator as ${\mathcal{A}}_{u}=\widetilde{\mathcal{L}}_{u}+\widetilde{\mathcal{I}}$, where

[TABLE]

Let $D$ be a bounded domain with $C^{1,1}$ boundary. Recall that $\uptau(D)$ denotes the first exit time from $D$. As shown in [ACPZ19]*Lemma 4.1, for any $f\in{\mathscr{W}}_{\text{loc}}^{2,d}(\mathds{R}^{d})$, such that $\widetilde{{\mathcal{I}}}\lvert f\rvert\in{L}_{\text{loc}}^{d}(\mathds{R}^{d})$, we have

[TABLE]

for all $x\in D$, $t\geq 0$, and $U\in{\mathfrak{U}}$. In addition, Eq. 2.5 holds if we replace $t\wedge\uptau(D)$ with $\uptau(D)$. Here, $\operatorname{\mathbb{E}}^{U}_{x}$ denotes the expectation operator on the canonical space of the process under the control $U\in{\mathfrak{U}}$. Equation Eq. 2.5 arises from the well known Krylov’s extension of the Itô’s formula, and we refer to this plainly as the Itô formula.

2.1. The ergodic control problem

Given a continuous running cost function ${\mathscr{R}}\colon\mathds{R}^{d}\times{\mathbb{U}}\rightarrow\mathds{R}_{+}$, which is locally Lipschitz continuous in its first argument uniformly with respect to the second, we define the average (or ergodic) penalty as

[TABLE]

for an admissible control $U\in{\mathfrak{U}}$. We say that $U\in{\mathfrak{U}}$ is stabilizing if $\varrho^{\vphantom{\frac{1}{2}}}_{U}(x)<\infty$ for all $x\in{\mathds{R}^{d}}$.

The ergodic control problem seeks to minimize the ergodic penalty over all admissible controls. We define

[TABLE]

As we show in Theorem 4.1, the optimal ergodic value $\varrho_{*}$ does not depend on $x$.

Assumption 2.1 which follows, is a slight variation of [ABP15]*Assumption 3.1, and is abstracted from the limiting diffusions arising in multiclass stochastic networks in the Halfin–Whitt regime. Note that the assumption on the running cost in [ACPZ19]*Section 2.2 is not met in these problems. Recall that a function $f\,\colon\mathcal{X}\to\mathds{R}$, where $\mathcal{X}$ is a $\sigma$-compact space, is called coercive, or inf-compact if the set $\{x\in\mathcal{X}\,\colon f(x)\leq C\}$ is compact (or empty) for every $C\in\mathds{R}$.

Assumption 2.1**.**

There exist some open set ${\mathcal{K}}\subset\mathds{R}^{d}$, a ball ${\mathscr{B}}_{\circ}$, and coercive nonnegative functions $\mathscr{V}_{\!\circ}\in\mathcal{C}^{2}(\mathds{R}^{d})$ and $F\in\mathcal{C}(\mathds{R}^{d}\times{\mathbb{U}})$ such that:

1. (i)

The running cost ${\mathscr{R}}$ is coercive on ${\mathcal{K}}$. 2. (ii)

The following inequalities hold

[TABLE]

Without loss of generality, we assume $F$ is locally Lipschitz continuous in its first argument.

Since we can always scale ${\mathscr{B}}_{\circ}$, $\mathscr{V}_{\!\circ}$ and $F$ to obtain the form in Eq. 2.8, there is no need to include any other constants in these equations. It is worth noting that ${\mathscr{R}}$ is coercive on ${\mathds{R}^{d}}$ if ${\mathcal{K}}^{c}$ is bounded, and the controlled jump diffusion is uniformly stable if ${\mathcal{K}}$ is bounded.

We introduce an additional assumption which, together with Assumption 2.1, is sufficient for the existence of a stabilizing stationary Markov control. For $v\in{{\mathfrak{U}}_{\mathrm{sm}}}$, we let b_{v}(x):=b\bigl{(}x,v(x)\bigr{)}, and define ${\mathcal{A}}_{v}$, $\widetilde{\mathcal{L}}_{v}$, ${\mathscr{R}}_{v}$, and $\varrho_{v}$ analogously. If under $v\in{{\mathfrak{U}}_{\mathrm{sm}}}$ the controlled jump diffusion is positive recurrent, then $v$ is called a stable Markov control, and the set of such controls is denoted by ${{\mathfrak{U}}_{\mathrm{ssm}}}$.

Assumption 2.2**.**

There exist $\hat{v}\in{{\mathfrak{U}}_{\mathrm{ssm}}}$, a positive constant $\hat{\kappa}$, and a coercive nonnegative function $\mathscr{V}\in\mathcal{C}^{2}(\mathds{R}^{d})$ such that

[TABLE]

with ${\mathscr{B}}_{\circ}$ as in Assumption 2.1.

Without loss of generality, we use the same ball ${\mathscr{B}}_{\circ}$ in Assumptions 2.1 and 2.2 in the interest of notational economy.

Remark 2.1*.*

The reader will note that Assumption 2.2 is not used in [ABP15]. Instead, starting from a weak stabilizability hypothesis, namely that

[TABLE]

the existence of a control $\hat{v}\in{{\mathfrak{U}}_{\mathrm{ssm}}}$ and a coercive nonnegative function $\mathscr{V}\in\mathcal{C}^{2}(\mathds{R}^{d})$ satisfying Eq. 2.9 is established in [ABP15]*Lemma 3.1. For the model studied in this paper, if we assume Eq. 2.10, then together with Assumption 2.1 we can show, that there exists a control $\hat{v}$ which is stabilizing for some coercive running cost $\widetilde{\mathscr{R}}\geq{\mathscr{R}}$ (see the proof of Theorem 4.1 which appears later). Then, if $\nu$ has compact support, [ACPZ19]*Theorem 3.7 shows that there exists a function $\mathscr{V}\in{\mathscr{W}}_{\text{loc}}^{2,p}(\mathds{R}^{d})$, for any $p>1$, satisfying Assumption 2.2, and this implies that $\widetilde{\mathcal{I}}\mathscr{V}\in{L}_{\text{loc}}^{d}(\mathds{R}^{d})$. Thus, if $\nu$ has compact support, then the Itô formula in Eq. 2.5 is applicable to $\mathscr{V}$, and using this in the proofs, it follows that as far as the results of this paper are concerned, we may replace Assumption 2.2 with the weaker hypothesis in Eq. 2.10, which cannot be weakened further since it is necessary for the value of the ergodic control problem to be finite. In typical applications, the existence of a stabilizing Markov control is usually established by exhibiting a Foster–Lyapunov equation taking the form of Eq. 2.9.

As we establish in Theorem 4.1, Assumption 2.1 and Eq. 2.10 together guarantee the existence of an optimal stationary Markov control for the ergodic control problem. Thus Assumption 2.2 need not be used for the existence part. However, it plays a crucial role in the derivation of the HJB equation in Section 5 for non-compactly supported $\nu$.

3. Examples

In this section, we provide examples of stochastic networks, and show that the jump diffusions involved satisfy Assumptions 2.1 and 2.2. We refer the reader to [AP16a, Section 2] for a detailed description of multiclass multi-pool networks.

Consider a multiclass multi-pool network with $d$ classes of customers and $J$ server pools. Define the sets ${\mathscr{I}}:=\{1,\dots,d\}$, ${\mathscr{J}}:=\{1,\dots,J\}$, and

[TABLE]

Following similar arguments as in [APZ19b, Theorem 2.1], and assuming that service interruptions are asymptotically negligible under the $\sqrt{n}$-scaling, we can show that the limiting controlled queueing processes are $d$-dimensional jump diffusions taking the form

[TABLE]

where $\sigma$ is a nonsingular diagonal matrix, $\theta$ is a strictly positive vector, and ${\{L_{t}\}_{t\geq 0}}$ is a one-dimensional compound Poisson process. The Lévy measure of $\theta L_{t}$ is denoted by $\nu(\mathrm{d}{z})$. This is supported on $\{\theta t\colon t\in[0,\infty)\}$. It follows by [AP16a, Lemma 4.3] that

[TABLE]

where $\ell\in\mathds{R}^{d}$, $\varGamma=\operatorname*{diag}(\gamma_{1},\dots,\gamma_{d})$, $M_{1}$ is a lower-diagonal $d\times d$ matrix with positive diagonal elements, and $M_{2}$ is a $d\times J$ matrix. Without loss of generality, we assume that $\gamma_{1}=0$, $\gamma_{d}>0$, and $\gamma_{i}\geq 0$, $i\in{\mathscr{I}}\setminus\{1,d\}$. We consider the ergodic control problem in Eq. 2.7 with

[TABLE]

for some $m\geq 1$, and some positive constants $\{c_{i}\colon i\in{\mathscr{I}}\}$ and $\{s_{j}\colon j\in{\mathscr{J}}\}$. This running cost function penalizes the queue sizes and idleness. It is evident that ${\mathscr{R}}(x,u)$ is not near-monotone, since $\langle e,x\rangle$ equals [math] on a hyperplane in $\mathds{R}^{d}$. We assume that $\int_{\mathds{R}^{d}}\lvert z\rvert^{m}\,\nu(\mathrm{d}{z})<\infty$.

We define ${\mathcal{K}}_{\delta}\,:=\,\{x\in\mathds{R}^{d}\colon\lvert\langle e,x\rangle\rvert>\delta\lvert x\rvert\}$ with $\delta>0$. It is clear that ${\mathscr{R}}$ is coercive on ${\mathcal{K}}_{\delta}$ for $\delta>0$. For a positive definite symmetric matrix $Q$, we let $g(x)$ be some positive convex smooth function which agrees with $\langle x,Qx\rangle^{\nicefrac{{1}}{{2}}}$ on $B_{1}^{c}$, and define the function \mathscr{V}_{Q,k}(x)=\bigl{(}g(x)\bigr{)}^{k} for $k>0$.

Lemma 3.1**.**

There exist a diagonal matrix $Q$, some $\delta>0$ small enough, and a positive constant $C$ such that $\mathscr{V}_{\circ}=\mathscr{V}_{Q,m}$ and $F(x)=C\lvert x\rvert^{m}$ satisfy Assumption 2.1 with ${\mathcal{K}}={\mathcal{K}}_{\delta}$.

Proof.

Recall $\widetilde{b}$ defined in Eq. 2.3. Following the same calculation as in the proof of [AP16a, Theorem 4.1], we obtain

[TABLE]

for some $\delta>0$, a positive constant $C_{1}$, and a diagonal matrix $Q$ satisfying $x^{\mathsf{T}}(QM_{1}+M_{1}^{\mathsf{T}}Q)x\geq 8\lvert x\rvert$. On the other hand, using the hypothesis $\int_{\mathds{R}^{d}}\lvert z\rvert^{m}\,\nu(\mathrm{d}{z})<\infty$, we obtain

[TABLE]

for some $\epsilon>0$ sufficiently small, and a positive constant $C_{2}$. Thus Eq. 2.8 holds. This completes the proof. ∎

Remark 3.1*.*

Let $\tilde{\ell}:=\ell+\int_{R^{d}}z\,\nu(\mathrm{d}{z})$ and $u^{c}_{1}=1$, and suppose that $\langle e,(M_{1}^{-1})^{\mathsf{T}}\tilde{\ell}\rangle>0$. Using the leaf elimination algorithm as in [AP16a, Theorem 4.2], we obtain a constant control $\bar{u}=(\bar{u}^{c},\bar{u}^{s})\in{\mathbb{U}}$, with $\bar{u}^{c}_{1}=1$, such that the last two terms on the right hand side of Eq. 3.2 are equal to [math]. This implies that ${\{X_{t}\}_{t\geq 0}}$ is transient under the control $\bar{u}$ by [APS19, Theorem 3.1]. Therefore, Eq. 3.1 is not uniformly stable.

Recall $\widetilde{\mathcal{L}}$ and $\widetilde{{\mathcal{I}}}$ defined in Eq. 2.4. By [AP16a, Theorem 4.2] concerning the local operator $\widetilde{\mathcal{L}}$, and Eq. 3.4 for $\widetilde{{\mathcal{I}}}$, it follows that there exist $u=(u^{c},u^{s})\in{\mathbb{U}}$ with $u^{c}_{d}=1$, and $\mathscr{V}(x)\sim\langle x,\tilde{Q}x\rangle^{\nicefrac{{m}}{{2}}}$ for some diagonal positive matrix $\tilde{Q}$ satisfying Assumption 2.2.

We present two specific examples: the ‘W’ and ‘V’ networks.

Example 3.1**.**

(The ‘W’ model with service interruptions.) See [AP16a, Section 4.2] for the detailed definition of the ‘W’ model. We have ${\mathscr{I}}=\{1,2,3\}$ and ${\mathscr{J}}=\{1,2\}$. By [AP16a, Example 4.2], $M_{1}$ and $M_{2}$ in Eq. 3.2 are given by

[TABLE]

for some positive constants $\{\mu_{ij}\colon i\in{\mathscr{I}},j\in{\mathscr{J}},(i,j)\notin\{(1,2),(3,1)\}\}$. We assume that $\gamma_{1}=\gamma_{2}=0$ and $\gamma_{3}=1$, and $\langle e,(M_{1}^{-1})^{\mathsf{T}}\tilde{\ell}\rangle>0$. By [AHP19, Theorem 3.1], under any control $v\in{{\mathfrak{U}}_{\mathrm{sm}}}$ with $v_{3}=0$ and $v_{5}=1$, $\{X_{t}\}_{t\geq 0}$ is transient. On the other hand, Assumption 2.2 is satisfied for the constant control $u^{c}_{3}=1$ and $u^{s}_{2}=1$.

Example 3.2**.**

(The ‘V’ model with service interruptions.) Equation 3.1 also describes the limiting jump diffusions of the ‘V’ model. Here ${\mathscr{I}}=\{1,\dots,d\}$, ${\mathscr{J}}=\{1\}$, and

[TABLE]

where $u$ takes values ${\mathbb{U}}=\{u\in\mathds{R}^{d}\colon\langle e,u\rangle=1\}$, and $M=\operatorname*{diag}(\mu_{1},\cdots,\mu_{d})$ is a positive diagonal matrix. Suppose that there exists a nonempty set ${\mathscr{I}}_{0}\subset\{1,\cdots,d-1\}$ such that $\gamma_{i}=0$ for $i\in{\mathscr{I}}_{0}$, and $\langle e,M^{-1}\tilde{\ell}\rangle>0$. In this case, [APS19, Theorem 3.3] asserts that ${\{X_{t}\}_{t\geq 0}}$ is transient under any $v\in{{\mathfrak{U}}_{\mathrm{ssm}}}$ satisfying $\varGamma v=0$. However, Assumption 2.1 is satisfied by [APS19, Remark 5.1], and, provided that $\gamma_{i}>0$ for some $i\in{\mathscr{I}}$, then Assumption 2.2 holds by [APS19, Theorem 3.5].

Remark 3.2*.*

It is shown in [AHP18] that the limiting diffusion of the ‘V’ model without service interruptions is uniformly ergodic over all stationary Markov controls, if either $\varGamma>0$, or the spare capacity $-\langle e,M^{-1}\ell\rangle$ is positive. This result has been extended to the limiting jump diffusion of the ‘V’ model with service interruptions in [AHPS19], with the difference that uniform ergodicity is over all stationary Markov controls resulting in a locally Lipschitz continuous drift. It is also shown in [AHP19] that if the spare capacity is positive, then the limiting diffusion of the multiclass multi-pool networks with a dominant server pool (for example the ‘N’ and ‘M’ models), or class-dependent service rates, is uniformly exponentially ergodic over all stationary Markov controls. However, in general, multiclass multi-pool networks do not enjoy uniform ergodicity, but fall in the framework of Assumptions 2.1 and 2.2.

4. Existence of an optimal stationary Markov control

In this section we establish the existence of an optimal stationary Markov control by following a standard convex analytic argument. We adopt the relaxed control framework (see, e.g., [ABG12]*Section 2.3), and extend the definitions of $b$ and ${\mathscr{R}}$ accordingly, that is we let $b_{v}(x)=\int_{{\mathbb{U}}}b(x,u)\,v(\mathrm{d}{u}\,|\,x)$, where $v(x)=v(\mathrm{d}{u}\,|\,x)$ is a measurable kernel on ${\mathbb{U}}$ given $x$, and analogously for ${\mathscr{R}}$. Let $\mu_{v}\in{\mathcal{P}}(\mathds{R}^{d})$ denote the unique invariant probability measure of Eq. 2.1 under $v\in{{\mathfrak{U}}_{\mathrm{ssm}}}$. Define the corresponding ergodic occupation measure $\uppi_{v}\in{\mathcal{P}}(\mathds{R}^{d}\times{\mathbb{U}})$ by $\uppi_{v}(\mathrm{d}{x},\mathrm{d}{u}):=\mu_{v}(\mathrm{d}{x})\,v(\mathrm{d}{u}\,|\,x)$. The class of all ergodic occupation measures is denoted by ${\mathscr{G}}$. Let $\mathcal{C}_{0}^{2}(\mathds{R}^{d})$ denote the Banach space of functions $f\colon{\mathds{R}^{d}}\to\mathds{R}$ that are twice continuously differentiable and their derivatives up to second order vanish at infinity, and ${\mathscr{C}}$ denote some fixed dense subset of $\mathcal{C}_{0}^{2}(\mathds{R}^{d})$ consisting of functions with compact supports. Applying the Theorem in [PE82], it follows that $\uppi\in{\mathscr{G}}$ if and only if

[TABLE]

It is easy to show that ${\mathscr{G}}$ is a closed and convex subset of ${\mathcal{P}}(\mathds{R}^{d}\times{\mathbb{U}})$ (see, e.g., [ABG12]*Lemma 3.2.3).

Recall also the definition of empirical measures.

Definition 4.1**.**

For $U\in{\mathfrak{U}}$ and $x\in\mathds{R}^{d}$, we define the mean empirical measures $\{\bar{\zeta}^{U}_{x,t}\,\colon t>0\}$, and (random) empirical measures $\{\zeta^{U}_{t}\,\colon t>0\}$ by

[TABLE]

and

[TABLE]

respectively, for all $f\in\mathcal{C}_{b}(\mathds{R}^{d}\times{\mathbb{U}})$.

Let $\overline{\mathds{R}}^{d}$ denote the one-point compactification of $\mathds{R}^{d}$. Then as shown in [ACPZ19]*Lemma 4.2, every limit $\hat{\zeta}\in{\mathcal{P}}(\overline{\mathds{R}}^{d}\times{\mathbb{U}})$ of $\zeta^{Z}_{x,t}$ as $t\to\infty$ takes the form $\hat{\zeta}=\delta\zeta^{\prime}+(1-\delta)\zeta^{\prime\prime}$ for some $\delta\in[0,1]$, with $\zeta^{\prime}\in{\mathscr{G}}$ and $\zeta^{\prime\prime}(\{\infty\}\times{\mathbb{U}})=1$ almost surely. The same claim holds for the mean empirical measures, without the qualifier ‘almost surely’.

We borrow the technique introduced in [ABP15]. Recall the function $F$ and the set ${\mathcal{K}}$ in Assumption 2.1. First, define the set

[TABLE]

We have

[TABLE]

As shown in [ABP15]*Lemma 3.3, there exists a coercive function $\widetilde{F}\in\mathcal{C}(\mathds{R}^{d}\times{\mathbb{U}})$, which is locally Lipschitz in its first argument, and satisfies

[TABLE]

for some positive constant $\tilde{\kappa}\geq 1$. Here again we select the same ball ${\mathscr{B}}_{\circ}$ as in Assumption 2.1 for convenience. This can always be accomplished by adjusting the constant $\tilde{\kappa}$.

Define the perturbed running cost ${\mathscr{R}}^{\epsilon}:={\mathscr{R}}+\epsilon\widetilde{F}$. Since ${\mathscr{R}}^{\epsilon}$ is coercive for $\epsilon>0$, the results of [ACPZ19] are applicable for the ergodic control problem with the perturbed running cost. At the same time, it follows from Eqs. 4.1 and 4.2 and the argument in the proof of [ABP15, Theorem 3.1], that if a control $U\in{\mathfrak{U}}$ is stabilizing for ${\mathscr{R}}$, then it is also stabilizing for ${\mathscr{R}}^{\epsilon}$ for any $\epsilon>0$.

Theorem 4.1**.**

Grant Assumption 2.1. Then every stabilizing stationary Markov control is in ${{\mathfrak{U}}_{\mathrm{ssm}}}$. In addition, if the stabilizability hypothesis in Eq. 2.10 is met, then there exists a stationary Markov control which is optimal for the ergodic control problem, and $\varrho_{*}$ is a constant.

Proof.

By Assumption 2.1, we have $\widetilde{{\mathcal{I}}}\mathscr{V}_{\circ}\in{L}_{\text{loc}}^{\infty}(\mathds{R}^{d})$, and thus, applying Itô’s formula and Fatou’s lemma to Eq. 4.1, it follows by Eq. 4.2 that

[TABLE]

Since by Eq. 4.3 we have

[TABLE]

for any stabilizing stationary Markov control $v$, we have $\uppi_{v}({\mathscr{R}})<\infty$, and the first assertion follows.

Define $\varrho^{\epsilon}_{U}$ and $\varrho^{\epsilon}_{*}$ as in Eqs. 2.6 and 2.7, respectively, by replacing ${\mathscr{R}}$ with ${{\mathscr{R}}}^{\epsilon}$. Let $\hat{\varrho}^{\epsilon}_{*}:=\inf_{\uppi\in{\mathscr{G}}}\,\uppi({\mathscr{R}}^{\epsilon})$, and $\hat{\varrho}_{*}:=\inf_{\uppi\in{\mathscr{G}}}\,\uppi({\mathscr{R}})$. Since ${\mathscr{R}}^{\epsilon}$ is coercive for any $\epsilon\in(0,1)$, we have $\hat{\varrho}^{\epsilon}_{*}=\uppi_{v_{*}^{\epsilon}}({\mathscr{R}}^{\epsilon})$ for some $v_{*}^{\epsilon}\in{{\mathfrak{U}}_{\mathrm{ssm}}}$ by [ABG12]*Theorem 3.4.5, and $\hat{\varrho}^{\epsilon}_{*}=\varrho^{\epsilon}_{*}$ by Lemma 4.2 in [ACPZ19] and the proof of [ABG12]*Theorem 3.4.7. Hence, by Eq. 4.3, which implies that $\varrho^{\epsilon}_{U}\,\leq\,\varrho_{U}+\epsilon\tilde{\kappa}(1+2\varrho_{U})$, and the above definitions we have

[TABLE]

This shows that $\varrho_{*}=\hat{\varrho}_{*}$. It remains to show that $\varrho_{*}=\uppi_{v_{*}}({\mathscr{R}})$ for some $v_{*}\in{{\mathfrak{U}}_{\mathrm{ssm}}}$. But this follows by using the technique in the proof of [ABG12]*Theorem 3.4.5. This completes the proof. ∎

5. The HJB equations

In this section, we study the $\alpha$-discounted and ergodic HJB equations for the jump diffusion defined in Eq. 2.1. For the $\alpha$-discounted control problem, it is rather standard to establish the existence of solutions and the characterization of optimal controls (see Theorem 5.1 below for details). We consider the Dirichlet problem on $B_{R}$ for the $\alpha$-discounted problem with running cost ${\mathscr{R}}^{\epsilon}$. From [BenLi-84]*Chap. 3, Theorem 2.3 and Remark 2.3, there exists a unique solution $\psi^{\epsilon}_{\alpha,R}\in{\mathscr{W}}^{2,p}(B_{R})\cap{\mathscr{W}}^{1,p}_{0}(B_{R})$ to the (homogeneous) Dirichlet problem

[TABLE]

For the Dirichlet problem with a linear integro-differential operator, existence and uniqueness of a solution are also asserted in [GM02]Theorem 3.1.22. Meanwhile, for a bounded running cost function [Menaldi-99](1.26) and under the blanket stability assumption in [Menaldi-99]*(1.6), HJB equations on the whole space are established in [Menaldi-99]Remark 3.3 and Theorem 4.1. It is clear that these assumptions are not met for multiclass stochastic networks in the Halfin–Whitt regime. For example, in Eq. 3.3, the running cost function penalizing the queueing and idleness is unbounded, and the drift in Eq. 3.1 does not satisfy [Menaldi-99](1.6).

Theorem 5.1**.**

Grant Assumptions 2.1 and 2.2. Then for any $\alpha\in(0,1)$ and $\epsilon\in[0,\tilde{\kappa}^{-1})$, the function $\psi^{\epsilon}_{\alpha,R}$ in Eq. 5.1 converges uniformly on compacta to a function $V^{\epsilon}_{\alpha}\in{\mathscr{W}}_{\text{loc}}^{2,p}(\mathds{R}^{d})$ for any $p>1$, which is the minimal nonnegative solution of the HJB equation

[TABLE]

and has the stochastic representation

[TABLE]

In addition, a control $v\in{{\mathfrak{U}}_{\mathrm{sm}}}$ is optimal, that is, it attains the infimum in Eq. 5.3, if and only if it is an a.e. measurable selector from the minimizer of Eq. 5.2.

Proof.

Under Assumption 2.2, the proof for the existence of a minimal nonnegative solution $V^{\epsilon}_{\alpha}\in{\mathscr{W}}_{\text{loc}}^{2,p}(\mathds{R}^{d})$ is exactly same as in [ACPZ19]*Theorem 3.2. A straightforward application of the comparison principle shows that the following bound holds

[TABLE]

From Eq. 5.4, we have $\widetilde{{\mathcal{I}}}V^{\epsilon}_{\alpha}\in{L}_{\text{loc}}^{\infty}(\mathds{R}^{d})$. Thus using the Itô’s formula in Eq. 2.5, the stochastic representation and the sufficiency part of the verification of optimality are established in a standard manner (see, e.g., [ABG12]*Theorem 3.5.6 and Remark 3.5.8). On the other hand, for any $v\in{{\mathfrak{U}}_{\mathrm{sm}}}$, the resolvent of the controlled diffusion defined in Eq. 2.2 has a positive density with respect to the Lebesgue measure by [ABG12]*Theorem A.3.5. Since the Lévy measure $\nu$ is finite, then applying [Li03]*Lemma 2.1, we see that the same holds for the resolvent of the jump diffusion in Eq. 2.1. Thus, we may repeat the argument in [ABG12]*Theorem 3.5.6 to establish the necessity part of the verification of optimality. This completes the proof. ∎

We proceed to derive the HJB equation on the ergodic control problem by using the vanishing discount method. The technique used has some important differences from [ACPZ19], since here the running cost is not near-monotone when $\epsilon=0$. To overcome this difficulty, we derive lower and upper bounds for $V^{\epsilon}_{\alpha}$ in the lemma which follows.

Lemma 5.1**.**

Grant the hypotheses in Assumptions 2.1 and 2.2. For any $\delta\in(0,\frac{1}{2}]$, there exists $\tilde{r}=r(\delta)>0$ such that

[TABLE]

for all $\alpha\in(0,1)$ and $\epsilon\in[0,\tilde{\kappa}^{-1})$. Moreover, there exists $r_{\circ}>0$ such that

[TABLE]

for all $\alpha\in(0,1)$ and $\epsilon\in[0,\tilde{\kappa}^{-1})$.

Proof.

Let $v_{*}$ be an optimal control in ${{\mathfrak{U}}_{\mathrm{ssm}}}$. Its existence has been asserted in Theorem 4.1. Recall that $\mu_{v_{*}}$ denotes the invariant probability measure under $v_{*}$. Using Eq. 5.3, Fubini’s theorem, and Eq. 4.4, we obtain

[TABLE]

for any $r>0$. Fix some $r_{\circ}>0$ such that $B_{r_{\circ}}\supset{\mathscr{B}}_{\circ}$. Then

[TABLE]

for all $r>r_{\circ}$, $\alpha\in(0,1)$, and $\epsilon\in[0,\tilde{\kappa}^{-1})$.

We first establish a lower bound of $V^{\epsilon}_{\alpha}$. Let $\psi^{\epsilon}_{\alpha,R}$ satisfy Eq. 5.1, and $\hat{v}_{R}\in{\mathfrak{U}}$ be a measurable selector from its minimizer, that is, it satisfies

[TABLE]

Let \delta\in\bigl{(}0,\frac{1}{2}\bigr{]} be arbitrary. By Eq. 5.7, and the coerciveness of $\widetilde{F}$ in Eq. 4.2, there exists $\tilde{r}=\tilde{r}(\delta)>r_{\circ}$ such that

[TABLE]

Let

[TABLE]

By Eqs. 4.2, 4.1, 5.8, and 5.9, we have

[TABLE]

Since $\psi^{\epsilon}_{\alpha,R}$ converges monotonically to $V^{\epsilon}_{\alpha}$ as $R\rightarrow\infty$ and $\mathscr{V}_{\!\circ}$ is coercive, there exists $R_{0}=R_{0}(\delta,\alpha)>\tilde{r}$ such that

[TABLE]

Thus, since $\phi^{\epsilon}_{\alpha,R}\geq 0$ on $B_{\tilde{r}}$ by Eq. 5.10, and $\phi^{\epsilon}_{\alpha,R}\geq 0$ on $B_{R}\setminus B_{R_{0}}$ by Eq. 5.12, it follows that $\phi^{\epsilon}_{\alpha,R}\geq 0$ on ${\mathds{R}^{d}}$ for all $R>R_{0}$ by Eq. 5.11 and the strong maximum principle. Taking limits as $R\rightarrow\infty$ in Eq. 5.10, we obtain

[TABLE]

which establishes Eq. 5.5.

Next we prove the upper bound. For $\hat{v}$ in Assumption 2.2, we have

[TABLE]

Recall that $B_{r_{\circ}}\supset{\mathscr{B}}_{\circ}$, and select any balls $D_{1}$ and $D_{2}$, such that $B_{r_{\circ}}\Subset D_{1}\Subset D_{2}$. By Eqs. 5.8, 4.2, 4.1, and 5.13, the function

[TABLE]

satisfies

[TABLE]

for all $\alpha\in(0,1)$ and $\epsilon\in[0,\tilde{\kappa}^{-1})$. It is evident that $\hat{\varphi}^{\epsilon}_{\alpha,R}\geq 0$ on $B_{r_{\circ}}\cup B_{R}^{c}$. Thus, employing the strong maximum principle, we obtain

[TABLE]

for all $\alpha\in(0,1)$ and $\epsilon\in[0,\tilde{\kappa}^{-1})$. Letting $R\rightarrow\infty$ in Eq. 5.14, we obtain Eq. 5.6. This completes the proof. ∎

We also need the following estimate. Its proof combines the technique in the proof of [ACPZ19]*Theorem 3.3 with Lemma 5.1.

Lemma 5.2**.**

Grant the hypotheses in Assumptions 2.1 and 2.2. For each $R>0$, there exists a constant $\kappa_{R}$ such that

[TABLE]

for all $\alpha\in(0,1)$ and $\epsilon\in[0,\tilde{\kappa}^{-1})$.

Proof.

We choose $B_{r_{\circ}}$, $D_{1}$, and $D_{2}$ as in the proof of Lemma 5.1. By Eqs. 2.9 and 4.1, it is evident that $\widetilde{{\mathcal{I}}}(\mathscr{V}_{\!\circ}+3{\mathscr{V}})\in{L}_{\text{loc}}^{\infty}(\mathds{R}^{d})$. Let $\hat{x}^{\epsilon}_{\alpha}\in\operatorname*{Arg\,min}_{\bar{D}_{2}}\,V^{\epsilon}_{\alpha}$. The function $\varphi^{\epsilon}_{\alpha}:=V^{\epsilon}_{\alpha}-V^{\epsilon}_{\alpha}(\hat{x}^{\epsilon}_{\alpha})$ satisfies

[TABLE]

where the inequality follows by Eq. 5.7. Using Eq. 5.6, we obtain

[TABLE]

$\alpha\in(0,1)$ and $\epsilon\in[0,\tilde{\kappa}^{-1})$. Let $v^{\epsilon}_{\alpha}$ be a measurable selector from the minimizer of the $\alpha$-discounted problem associated with ${\mathscr{R}}^{\epsilon}$. By the local maximum principle [GilTru]*Theorem 9.20, for any $p>0$, there exists a constant $\tilde{C}_{1}(p)>0$ such that

[TABLE]

with \lVert\varphi^{\epsilon}_{\alpha}\rVert_{p;D_{1}}:=\bigl{(}\int_{D_{1}}\lvert\varphi^{\epsilon}_{\alpha}(x)\rvert\mathrm{d}{x}\bigr{)}^{\nicefrac{{1}}{{p}}}, and by the supersolution estimate [GilTru]*Theorem 9.22, there exist some $p>0$ and $\tilde{C}_{2}>0$ such that

[TABLE]

On the other hand, the inequality in Eq. 5.5 implies that $\inf_{D_{1}}\,\varphi^{\epsilon}_{\alpha}\leq\sup_{D_{2}}\mathscr{V}_{\!\circ}$. Combining these estimates, we obtain

[TABLE]

where

[TABLE]

By Eqs. 5.15 and 5.16, we have

[TABLE]

Hence, either $\sup_{D_{2}}\,\varphi^{\epsilon}_{\alpha}\leq 2\kappa_{2}+2\lVert\mathscr{V}_{\!\circ}+3\mathscr{V}\rVert_{{L}^{\infty}(D_{2})}$, which directly implies Eq. 5.7, or

[TABLE]

Suppose that Eq. 5.17 is the case. By Eq. 5.15, we have the estimate

[TABLE]

Thus, by Eqs. 5.16, 5.17, and 5.18, we obtain

[TABLE]

Again we distinguish two cases. If

[TABLE]

then the proof is the same as in [ACPZ19]*Theorem 3.3. It remains to consider the case

[TABLE]

Let $\tilde{\phi}_{\alpha}^{\epsilon}$ be the solution of the Dirichlet problem

[TABLE]

By Harnack’s inequality, we have $\tilde{\phi}^{\epsilon}_{\alpha}\leq\tilde{C}_{H}\tilde{\phi}^{\epsilon}_{\alpha}(\hat{x}^{\epsilon}_{\alpha})$ for all $x\in B_{r_{\circ}}$, $\alpha\in(0,1)$, and $\epsilon\in[0,\tilde{\kappa}^{-1})$. Thus

[TABLE]

and $\varphi^{\epsilon}_{\alpha}-\tilde{\phi}^{\epsilon}_{\alpha}=0$ on $\partial D_{1}$. On the other hand, we have

[TABLE]

Using Eq. 5.5, we obtain

[TABLE]

Since $\widetilde{\mathcal{I}}\mathscr{V}_{\!\circ}\in{L}_{\text{loc}}^{\infty}(\mathds{R}^{d})$, applying the ABP weak maximum principle in [GilTru]*Theorem 9.1 to Eq. 5.20, and using Eqs. 5.19 and 5.21, we obtain $\lVert\varphi_{\alpha}^{\epsilon}-\tilde{\phi}^{\epsilon}_{\alpha}\rVert_{{L}^{\infty}(D_{1})}\leq\tilde{C}_{0}$ for some constant $\tilde{C}_{0}$ which does not depend on $\alpha\in(0,1)$ and $\epsilon\in[0,\tilde{\kappa}^{-1})$. Thus, employing [AA-Harnack]*Corollary 2.2 as done in [ACPZ19]*Theorem 3.3, we establish Eq. 5.7. This completes the proof. ∎

In Theorem 5.2 which follows, we derive the HJB equation for the ergodic control problem, and the corresponding characterization of optimal Markov controls. Compared to [ACPZ19, Theorem 4.5], the important difference here is that the solutions to the HJB equation may not be bounded from below in $\mathds{R}^{d}$, since the running cost function is not near-monotone. As a consequence, [ABG12, Lemma 3.6.9] cannot be applied here directly to establish the stochastic representation of the solutions, and prove uniqueness. We let V_{\alpha}:=\ V_{\alpha}^{\epsilon}\bigr{|}_{\epsilon=0}.

Theorem 5.2**.**

Grant Assumptions 2.1 and 2.2. Then

1. (a)

As $\alpha\searrow 0$, $\widetilde{V}_{\alpha}:=V_{\alpha}-V_{\alpha}(0)$ converges in $\mathcal{C}^{1,\rho}$ with $\rho\in(0,1)$, uniformly on compact sets, to a function $V_{*}\in{\mathscr{W}}_{\text{loc}}^{2,p}(\mathds{R}^{d})$ for any $p>1$, which satisfies $V^{-}_{*}\in{\mathfrak{o}}(\mathscr{V}_{\!\circ})$, and

[TABLE] 2. (b)

A control $v\in{{\mathfrak{U}}_{\mathrm{ssm}}}$ is optimal for the ergodic control problem with ${\mathscr{R}}$ if and only if it is an a.e. measurable selector from the minimizer in Eq. 5.22. 3. (c)

Let ${{\tilde{\mathfrak{U}}}_{\mathrm{ssm}}}:=\{v\in{{\mathfrak{U}}_{\mathrm{ssm}}}\colon\varrho_{v}<\infty\}$. The function $V_{*}$ is the unique solution (up to an additive constant) to the equation $\min_{u\in{\mathbb{U}}}\,[{\mathcal{A}}_{u}V_{*}(x)+{\mathscr{R}}(x,u)]=\varrho$ a.e. on ${\mathds{R}^{d}}$, with $\varrho\leq\varrho_{*}$, which satisfies $V^{-}_{*}\in{\mathfrak{o}}({\mathscr{V}_{\!\circ}})$ and $V_{*}(0)=0$. In addition, it has the stochastic representation

[TABLE]

Proof.

We first prove (a). By Eq. 5.2, we have

[TABLE]

The limit $\lim_{\alpha\searrow 0}\alpha V_{\alpha}(0)=\varrho_{*}$ follows as in the proof of Theorem 3.6 in [ABP15] using Lemma 5.2.

We fix an arbitrary ball ${\mathscr{B}}$, and using Eqs. 5.5 and 5.6, we obtain

[TABLE]

By Eqs. 5.25, 4.1, and 2.9, we have $\widetilde{{\mathcal{I}}}\lvert\widetilde{V}_{\alpha}\rvert\in{L}_{\text{loc}}^{\infty}(\mathds{R}^{d})$. Let $v_{\alpha}$ be a measurable selector for the minimizer of Eq. 5.24. Then, applying the interior estimate in [GilTru]*Theorem 9.11, we obtain

[TABLE]

where $C\equiv C(R,p)$. Hence, $\sup_{\alpha\in(0,1)}\lVert\widetilde{V}_{\alpha}\rVert_{{\mathscr{W}}^{2,p}(B_{R})}<\infty$. Thus, following a standard argument (see [ABG12]*Lemma 3.5.4), for any sequence $\alpha_{n}\searrow 0$, the functions $\{\widetilde{V}_{\alpha_{n}}\}$ converge along a subsequence in $\mathcal{C}^{1,\rho}$ with $\rho\in(0,1)$, uniformly on compact sets, to $V_{*}$. Since $R$ is arbitrary, this proves Eq. 5.22. Letting $\alpha\searrow 0$ in Eq. 5.5, which holds for all $\delta>0$, with $\tilde{r}$ depending only on $\delta$, we obtain (with $\epsilon=0$)

[TABLE]

Concerning part (b), necessity follows by [ACPZ19]*Theorem 3.5. Sufficiency follows exactly as in the proof of [ABP15]*Theorem 3.4 (b) using Eq. 5.26 and part (a).

It remains to establish uniqueness and the stochastic representation as stated in part (c). By Eq. 5.25, we also have $\widetilde{{\mathcal{I}}}V_{*}\in{L}_{\text{loc}}^{\infty}(\mathds{R}^{d})$. Following the same arguments as in [ABG12]*Lemma 3.6.9, we have

[TABLE]

On the other hand, applying Itô’s formula, we obtain

[TABLE]

with $v_{0}$ a measurable selector for the minimizer of Eq. 5.22. Note that

[TABLE]

We next show that

[TABLE]

Let $\Phi:=V_{*}+\mathscr{V}_{\!\circ}$. It follows by Eq. 5.26 that $\Phi$ is coercive and $V^{-}_{*}\in{\mathfrak{o}}(\Phi)$. It is also evident by Eqs. 4.1 and 5.22 that ${\mathcal{A}}_{v_{0}}\Phi(x)\leq\mathds{1}_{{\mathscr{B}}_{\circ}}(x)+\varrho_{*}$. Then, applying Itô’s formula, we obtain

[TABLE]

We also have

[TABLE]

Note that $\operatorname{\mathbb{E}}_{x}^{v_{0}}[{\breve{\uptau}}_{r}]$ is finite since $v_{0}\in{{\mathfrak{U}}_{\mathrm{ssm}}}$ by Theorem 4.1. Since $V_{*}^{-}\in{\mathfrak{o}}(\Phi)$, it follows that $\sup_{y\in B_{R}^{c}}\,\frac{V^{-}_{*}(y)}{\Phi(y)}$ vanishes as $R\to\infty$, which in turn implies that the right hand side of Eq. 5.30 converges to [math] as $R\to\infty$. This proves Eq. 5.29. Letting $R\to\infty$ in Eq. 5.28, it follows by Fatou’s lemma and Eq. 5.29 that

[TABLE]

and we obtain

[TABLE]

which together with Eq. 5.27 implies Eq. 5.23. Note that, by the argument above, Eq. 5.31 holds for any solution $V$ of Eq. 5.22 which satisfies $V^{-}\in{\mathfrak{o}}({\mathscr{V}_{\!\circ}})$. Thus, if $V$ is any other solution, we have $V_{*}\leq V$ and equality follows by the strong maximum principle. This completes the proof. ∎

5.1. Regularity of solutions of the HJB

In this section, we examine the regularity of solutions of the HJB equations in Theorems 5.1 and 5.2. If the Lévy measure $\nu$ has a compact support, then it follows by the elliptic regularity [GilTru, Theorem 9.19] that the solutions to the HJB equations are in $\mathcal{C}^{2,r}(\mathds{R}^{d})$ for any $r\in(0,1)$. See also Remark 3.4 in [ACPZ19].

We need the following gradient estimate which is also applicable to a larger class of equations.

Lemma 5.3**.**

Let $\varphi\in{\mathscr{W}}_{\text{loc}}^{2,p}(\mathds{R}^{d})$, $p>d$, be a strong solution, having at most polynomial growth of degree $m>0$, to the equation

[TABLE]

where

1. (i)

the matrix $a$ is bounded, Lipschitz continuous on $\mathds{R}^{d}$ and uniformly elliptic; 2. (ii)

the coefficients $b$ and $c$ are locally bounded and measurable, with $b$ having at most linear growth and $c$ having at most quadratic growth; 3. (iii)

the function $f$ has at most polynomial growth of degree $\kappa$ with $\kappa\in(0,m+2]$; 4. (iv)

the Lévy measure $\nu$ of the nonlocal operator $\widetilde{{\mathcal{I}}}$ is finite and satisfies $\int_{\mathds{R}^{d}}\lvert z\rvert^{m}\,\nu(\mathrm{d}{z})<\infty$.

Then, $\lvert\nabla{\varphi}(x)\rvert\in{\mathscr{O}}(\lvert x\rvert^{m+1})$.

Proof.

For any fixed $x_{0}\in\mathds{R}^{d}$, for which without loss of generality we assume $\lvert x_{0}\rvert\geq 1$, we define the scaled variables

[TABLE]

and similarly for $\tilde{a}$, $\tilde{b}$, $\tilde{c}$, and $\tilde{f}$. The equation in Eq. 5.32 then takes the form

[TABLE]

It is clear from (i)–(ii) that the coefficients $\tilde{a}^{ij}$, $\lvert x_{0}\rvert^{\nicefrac{{-1}}{{2}}}\tilde{b}^{i}$ and $\lvert x_{0}\rvert^{-1}\tilde{c}$ are bounded in the ball $B_{2}(x_{0})$, with a bound independent of $x_{0}$, and that the Lipschitz and ellipticity constants of the matrix $\tilde{a}$ in $B_{2}(x_{0})$ are independent of $x_{0}$. Thus, it follows by Eq. 5.33 and the a priori estimate in [GilTru]*Theorem 9.11 that, for any fixed $p>d$, we have

[TABLE]

for some positive constant $C$ independent of $x_{0}$. Since $\nu$ is finite and $\int_{\mathds{R}^{d}}\lvert z\rvert^{m}\,\nu(\mathrm{d}{z})<\infty$, it follows that $\widetilde{{\mathcal{I}}}\varphi$ has at most polynomial growth of degree $m$. Then, by the assumptions of the lemma, the right hand side of Eq. 5.34 is {\mathscr{O}}\bigl{(}\lvert x_{0}\rvert^{\nicefrac{{m}}{{2}}}\bigr{)}. Therefore, by Eq. 5.34 and the compactness of the Sobolev embedding {\mathscr{W}}^{2,p}\bigl{(}B_{1}(x_{0})\bigr{)}\hookrightarrow\mathcal{C}^{1,r}\bigl{(}B_{1}(x_{0})\bigr{)}, for $0<r<1-\frac{d}{p}$, we obtain

[TABLE]

for some positive constant $C_{0}$ independent of $x_{0}$. On the other hand,

[TABLE]

which together with Eq. 5.35 imply $\lvert\nabla\varphi(x)\rvert\in{\mathscr{O}}(\lvert x\rvert^{m+1})$. This completes the proof. ∎

Consider the following assumption on the growth of the coefficients and the functions $\mathscr{V}_{\!\circ}$ and $\mathscr{V}$.

Assumption 5.1**.**

1. (i)

The running cost function has at most polynomial growth of degree $m_{\circ}\geq 1$, that is, ${\mathscr{R}}(x,u)\leq C_{\circ}(1+\lvert x\rvert^{m_{\circ}})$, for all $(x,u)\in\mathds{R}^{d}\times{\mathbb{U}}$ and some positive constant $C_{\circ}$. 2. (ii)

Assumptions 2.1 and 2.2 hold with $\mathscr{V}$ and $\mathscr{V}_{\!\circ}$ having at most polynomial growth of degree $m_{\circ}$. 3. (iii)

The Lévy measure $\nu$ satisfies $\int_{\mathds{R}^{d}}\lvert z\rvert^{m_{\circ}+1}\,\mathrm{d}{z}<\infty$.

Remark 5.1*.*

Provided that $\int_{\mathds{R}^{d}}\lvert z\rvert^{m+1}\,\nu(\mathrm{d}{z})<\infty$, it is clear that Assumption 5.1 holds for the limiting controlled diffusion in Section 3.

We have the following theorem.

Theorem 5.3**.**

Grant Assumption 5.1. The solutions $V^{\epsilon}_{\alpha}$ of Eq. 5.2, and $V_{*}$ of Eq. 5.22 are in $\mathcal{C}^{2,r}(\mathds{R}^{d})$ for any $r\in(0,1)$.

Proof.

Consider $V_{*}$. Since $V_{*}\in{\mathscr{O}}(\mathscr{V}_{\!\circ}+3\mathscr{V})$ by Eq. 5.25, then $\widetilde{{\mathcal{I}}}V_{*}\in{L}_{\text{loc}}^{\infty}(\mathds{R}^{d})$ by Eqs. 2.8 and 2.9, and $V_{*}$ has at most polynomial growth of degree $m_{\circ}$ by Assumption 5.1. We claim that $\widetilde{{\mathcal{I}}}V_{*}$ is locally Lipschitz continuous. To prove the claim, we fix some $x_{0}\in{\mathds{R}^{d}}$, and write

[TABLE]

for all $x^{\prime},x^{\prime\prime}\in B_{1}$. By Eq. 5.22 and Lemma 5.3 we obtain

[TABLE]

which together with Assumption 5.1 (iii) and Eq. 5.36 proves the claim. It then follows by Eq. 5.22 and elliptic regularity (see [GilTru]*Theorem 9.19) that $V_{*}$ is in $\mathcal{C}^{2,r}(\mathds{R}^{d})$ for any $r\in(0,1)$. The proof of the same property for $V^{\epsilon}_{\alpha}$ is completely analogous. ∎

6. Pathwise optimality

The pathwise formulation of the ergodic control problem seeks to a.s. minimize over $U\in{\mathfrak{U}}$

[TABLE]

where ${\{X^{U}_{t}\}_{t\geq 0}}$ denotes the process governed by Eq. 2.1 under the control $U$. If the running cost is near-monotone or a uniform stability condition holds, it follows by [ACPZ19, Theorem 4.4] that every average cost optimal stationary Markov control is also optimal with respect to the pathwise ergodic criterion. In this section, we extend the results of the diffusion model in [AA17] to the jump-diffusion model. We modify Assumption 2.1 as follows.

Assumption 6.1**.**

Assumption 2.1 holds with F=\phi\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}\mathscr{V}_{\!\circ}, where $\phi\colon\mathds{R}_{+}\to\mathds{R}_{+}$ is smooth increasing concave function, satisfying $\phi(z)\to\infty$ as $z\to\infty$. In addition, the functions $\upsigma$ and \frac{\nabla\mathscr{V}_{\!\circ}}{1+\phi\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}\mathscr{V}_{\!\circ}} are bounded on ${\mathds{R}^{d}}$.

Remark 6.1*.*

For the examples in Section 3, we may rescale $F(x)$ in Lemma 3.1 so that $F(x)=C\mathscr{V}_{Q,m}(x)$ for some positive constant $C$. Thus, if we choose $\phi(x)=Cx$, then Assumption 6.1 holds.

Theorem 6.1**.**

Grant Assumption 6.1. Then, if $U\in{\mathfrak{U}}$ is such that $\varrho_{U}(x)<\infty$, the family of random empirical measures $\{\zeta^{U}_{t}\,\colon t>0\}$ in Definition 4.1 is tight a.s. In particular, every average cost optimal stationary Markov control is pathwise optimal.

Proof.

The technique is similar to that of [AA17]*Theorem 3.1, but here we need to account for the nonlocal term. We define $\psi_{N}\colon\mathds{R}_{+}\to\mathds{R}_{+}$, for $N\in\mathds{N}$, by

[TABLE]

and $\varphi_{N}\colon{\mathds{R}^{d}}\to\mathds{R}_{+}$ by \varphi_{N}:=\psi_{N}\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}\mathscr{V}_{\!\circ}, where ‘\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}’ denotes composition of functions. Let $U\in{\mathfrak{U}}$ be some admissible control such that

[TABLE]

for some increasing divergent sequence $\{t_{n}\}$. Since ${\mathscr{R}}$ is coercive on ${\mathcal{K}}\times{\mathbb{U}}$, it follows that $\zeta^{U}_{t_{n}}$ is a.s. tight when restricted to ${\mathfrak{B}}({\mathcal{K}}\times{\mathbb{U}})$, the Borel $\sigma$-algebra of ${\mathcal{K}}\times{\mathbb{U}}$.

Since $\nabla\varphi_{N}$ and $\upsigma$ are bounded, by Itô’s formula and Eq. 2.8 we obtain

[TABLE]

Then, the second and third terms on the right hand side of Eq. 6.1 converge to [math] a.s. as $t\to\infty$, by the proof of Lemma 4.2 in [ACPZ19].

An easy computation shows that

[TABLE]

where

[TABLE]

and

[TABLE]

To estimate Eq. 6.4, we use

[TABLE]

Then combining this with Eqs. 6.1, 6.2, 6.3, and 6.4, we obtain

[TABLE]

for some constant $C$, from which it follows that $\zeta^{U}_{t_{n}}$ is a.s. tight when restricted to ${\mathfrak{B}}({\mathcal{K}}^{c}\times{\mathbb{U}})$. Therefore, it is a.s. tight in ${\mathcal{P}}({\mathds{R}^{d}}\times{\mathbb{U}})$. This completes the proof. ∎

7. An approximate HJB equation

In this section, we use an approximate HJB equation to construct $\epsilon$-optimal controls. Its purpose is twofold. First, it is used to establish asymptotic optimality in [APZ19b]. Second, the approximating HJB equation is a semilinear equation on a sufficiently large ball, and a linear equation on its complement, which is beneficial to numerical methods. This result was first reported in [ABP15, Section 4] for a continuous diffusion. The proof in that paper crucially relied on the the following property of a positive recurrent nondegenerate diffusion of the form $\mathrm{d}X_{t}=b(X_{t})\mathrm{d}t+\upsigma(X_{t})\mathrm{d}W_{t}$: If a function $f\colon{\mathds{R}^{d}}\to\mathds{R}$ is integrable under the invariant probability measure $\mu$ of the diffusion, then $\operatorname{\mathbb{E}}[f(X_{t})]\to\mu(f)$ as $t\to\infty$. The proof of this property in [Ichihara-13, Proposition 2.6] relies on the Harnack property which we do not have for the model in this paper. Thus, a different approach is adopted.

We first consider the ergodic control problem with a suitable control which satisfies Assumption 2.2 fixed outside a ball of arbitrarily large radius. Then, we show that $\epsilon$-optimal controls are obtained by choosing the radius of the ball sufficiently large. Assumption 7.1 replaces Assumption 2.2 in this section.

Assumption 7.1**.**

The following hold:

1. (i)

The function $\widetilde{F}$ in Eq. 4.2 is in $\mathcal{C}^{2}({\mathds{R}^{d}})$, has at most polynomial growth of degree $\widetilde{m}\geq 1$, and satisfies

[TABLE] 2. (ii)

There exist $\tilde{v}\in{{\mathfrak{U}}_{\mathrm{ssm}}}$ and $\widetilde{\mathscr{V}}\in\mathcal{C}^{2}({\mathds{R}^{d}})$, with $\widetilde{\mathscr{V}}\in{\mathscr{O}}(\widetilde{F})$, satisfying

[TABLE]

for some positive constant $\widetilde{C}$; 3. (iii)

The Lévy measure $\nu$ satisfies $\int_{{\mathds{R}^{d}}}\lvert z\rvert^{\widetilde{m}}\,\nu(\mathrm{d}{z})<\infty$.

The jump diffusion with a ‘truncated’ control space is defined as following.

Definition 7.1**.**

With $\tilde{v}\in{{\mathfrak{U}}_{\mathrm{ssm}}}$ as in Assumption 7.1 and each $R>0$, we define

[TABLE]

Let ${\mathcal{A}}^{R}_{u}$ denote the operator associated with the controlled jump diffusion

[TABLE]

with $X_{0}=x\in\mathds{R}^{d}$, and define

[TABLE]

where ${{\mathfrak{U}}_{\mathrm{sm}}}(\tilde{v},R)$ denotes the class of stationary Markov controls which agree with $\tilde{v}\in{{\mathfrak{U}}_{\mathrm{ssm}}}$ on $B_{R}^{c}$.

By Theorem 5.2, for each $R>0$ and $\epsilon\in(0,1)$, there exists a unique $V^{R}_{\epsilon}\in{\mathscr{W}}_{\text{loc}}^{2,p}(\mathds{R}^{d})$, for any $p>1$, which is bounded from below in $\mathds{R}^{d}$ and satisfies $V^{R}_{\epsilon}(0)=0$, and

[TABLE]

In addition, there exists a constant $C$ such that

[TABLE]

It is clear that $\varrho^{\epsilon,R}_{*}$ is nonincreasing. Let $\hat{\varrho}^{\epsilon}:=\lim_{R\to\infty}\varrho^{\epsilon,R}_{*}$. As in the proof of Theorem 5.2, $V^{R}_{\epsilon}\rightarrow\widehat{V}_{\epsilon}\in{\mathscr{W}}_{\text{loc}}^{2,p}(\mathds{R}^{d})$, for any $p>1$, which satisfies

[TABLE]

Recall that ${\mathscr{R}}^{\epsilon}$, defined in the proof of Theorem 4.1, is the optimal ergodic value for the controlled diffusion in Eq. 2.1 with running cost ${\mathscr{R}}^{\epsilon}$. We wish to show that $\hat{\varrho}^{\epsilon}=\varrho^{\epsilon}_{*}$. To establish this we need the following lemma, which provides a lower bound for supersolutions of a general class of integro-differential equations.

Lemma 7.1**.**

Let $\varphi\in{\mathscr{W}}_{\text{loc}}^{2,d}(\mathds{R}^{d})$, be a supersolution of the equation

[TABLE]

which is bounded below in ${\mathds{R}^{d}}$. Assume the following:

1. (i)

the matrix $a$ is nonsingular and satisfies

[TABLE] 2. (ii)

the drift $b$ is measurable and has at most linear growth; 3. (iii)

the function $\widetilde{F}$ is as in Assumption 7.1; 4. (iv)

the operator ${\mathcal{I}}\colon\mathcal{C}^{1}(\mathds{R}^{d})\mapsto\mathcal{C}(\mathds{R}^{d})$ is given by

[TABLE]

for $h\in\mathcal{C}^{1}(\mathds{R}^{d})$ and the Lévy measure $\nu$ satisfies

[TABLE]

Then, $\widetilde{F}\in{\mathscr{O}}(\varphi)$.

Proof.

We have

[TABLE]

By using (i)–(iii), it is clear that ${\mathscr{A}}\widetilde{F}-{{\mathcal{I}}}\widetilde{F}\in{\mathscr{O}}(\widetilde{F})$. By (iii) and (iv), it follows by [APS19]*Lemma 5.1 that ${{\mathcal{I}}}\widetilde{F}\in{\mathscr{O}}(\widetilde{F})$. Thus, there exists $\hat{r}>0$ such that

[TABLE]

for some positive constant $C$. Let $\phi_{n}(x)$ be a smooth cutoff function satisfying $\phi_{n}(x)=1$ on $B_{n}$ and $\phi_{n}(x)=0$ on $B^{c}_{n+1}$, for $n\in\mathds{N}$. By Eq. 7.4 and (iii), we can choose $r>\hat{r}$ large enough and $\epsilon\in(0,1)$ sufficiently small so that for any $n\in\mathds{N}$,

[TABLE]

Let $M$ be a lower bound for $\varphi$, and $n>r$. We define the function $\hat{\varphi}^{n}_{\epsilon}(x):=\varphi(x)-\epsilon\widetilde{F}(x)\phi_{n}(x)-(M-\sup_{B_{r}}\widetilde{F})$. Then, applying Eqs. 7.3 and 7.5, we have

[TABLE]

It is evident that $\varphi(x)-(M-\sup_{B_{r}}\widetilde{F})\geq 0$ on $B^{c}_{n+1}$, and $\varphi(x)-\epsilon\widetilde{F}(x)-(M-\sup_{B_{r}}\widetilde{F})\geq 0$ on $B_{r}$. Thus, applying the strong maximum principle, we obtain $\hat{\varphi}^{n}_{\epsilon}(x)\geq 0$ in $\mathds{R}^{d}$. It follows that $\varphi(x)\geq\epsilon\widetilde{F}(x)\phi_{n}(x)+(M-\sup_{B_{r}}\widetilde{F})$ in $\mathds{R}^{d}$ for all $n$ large enough. This completes the proof. ∎

The main result of this section is the following.

Theorem 7.1**.**

Grant Assumption 7.1. Then, $\hat{\varrho}^{\epsilon}=\varrho^{\epsilon}_{*}$.

Proof.

Let $V_{\epsilon}\in{\mathscr{W}}_{\text{loc}}^{2,p}(\mathds{R}^{d})$, $p>1$, be the unique solution of the equation

[TABLE]

which is bounded below in ${\mathds{R}^{d}}$ and satisfies $V_{\epsilon}(0)=0$. Applying Itô’s formula to Eq. 7.6, we obtain

[TABLE]

with $v^{\epsilon}_{*}$ a measurable selector from the minimizer of Eq. 7.7. It is clear from Eq. 7.7 that G(T):=\lim_{r\to\infty}\operatorname{\mathbb{E}}^{v^{\epsilon}_{*}}_{x}\bigl{[}V_{\epsilon}(X_{T\wedge\uptau_{r}})\bigr{]} exists and satisfies $\limsup\frac{1}{T}G(T)\to 0$ by Birkhoff’s ergodic theorem. By Lemma 7.1, we have ${\mathscr{R}}^{\epsilon}_{v^{\epsilon}_{*}}\in{\mathscr{O}}(V_{\epsilon})$, and this implies that $\widehat{V}_{\epsilon}\in{\mathscr{O}}(V_{\epsilon})$ by Assumption 7.1 (ii) and Eq. 7.1. Therefore, if \widehat{G}(T):=\limsup_{r\to\infty}\operatorname{\mathbb{E}}^{v^{\epsilon}_{*}}_{x}\bigl{[}\widehat{V}_{\epsilon}(X_{T\wedge\uptau_{r}})\bigr{]}, then $\limsup_{T\to\infty}\frac{1}{T}\widehat{G}(T)\to 0$. Thus, evaluating Eq. 7.2 at $v^{\epsilon}_{*}$, and applying Itô’s formula, we obtain

[TABLE]

from which it follows that $\varrho^{\epsilon}_{*}\geq\hat{\varrho}^{\epsilon}$. This of course implies that $\varrho^{\epsilon}_{*}=\hat{\varrho}^{\epsilon}$, since Eq. 7.2 has no bounded from below solutions for $\hat{\varrho}^{\epsilon}<\varrho^{\epsilon}_{*}$. ∎

The following corollary concerns the construction of continuous precise $\epsilon$-optimal controls. It follows directly from Theorem 7.1 and the method in [AP16a, Theorem 5.5].

Corollary 7.1**.**

For any given $\epsilon>0$ and $\tilde{v}$ satisfying Assumption 2.2, there exist $R=R(\epsilon)>0$ and a continuous precise control $v_{\epsilon}\in{{\mathfrak{U}}_{\mathrm{ssm}}}$ such that $v_{\epsilon}\equiv\tilde{v}$ on $\bar{B}^{c}_{R}$, and

[TABLE]

Acknowledgments

This research was supported in part by the Army Research Office through grant W911NF-17-1-001, and in part by the National Science Foundation through grants DMS-1715210, CMMI-1538149 and DMS-1715875, and in part by Office of Naval Research through grant N00014-16-1-2956 and was approved for public release under DCN #43-5439-19.