A variational characterization of the risk-sensitive average reward for controlled diffusions on $\mathbb{R}^d$

TL;DR

This paper develops a variational framework for the risk-sensitive reward problem in controlled diffusions on bd, linking it to eigenvalues of associated operators and extending results to unbounded drifts and costs.

Contribution

It introduces a variational formula for the risk-sensitive value and connects it to the principal eigenvalue of a semilinear operator, extending previous results.

Findings

Established a variational formula on bd for the risk-sensitive reward.

Showed the risk-sensitive value equals the generalized principal eigenvalue.

Extended results to unbounded drifts and costs using a new gradient estimate.

Abstract

We address the variational formulation of the risk-sensitive reward problem for non-degenerate diffusions on $\mathbb{R}^d$ controlled through the drift. We establish a variational formula on the whole space and also show that the risk-sensitive value equals the generalized principal eigenvalue of the semilinear operator. This can be viewed as a controlled version of the variational formulas for principal eigenvalues of diffusion operators arising in large deviations. We also revisit the average risk-sensitive minimization problem and by employing a gradient estimate developed in this paper, we extend earlier results to unbounded drifts and running costs.

Equations

$$\mathrm{d}{X}_t=b(X_t,{\xi }_t)\mathrm{d}t+\text{upsigma}(X_t)\mathrm{d}{W}_t$$

$$J_{*}\,\coloneqq\,\sup_{\{\xi_{t}\}_{t\geq 0}}\;\liminf_{T\to\infty}\,\frac{1}{T}\,\log\operatorname{\mathbb{E}}\Bigl{[}\mathrm{e}^{\int^{T}_{0}c(X_{t},\xi_{t})\,\mathrm{d}t}\Bigr{]}\,,$$

$${\mathscr{A}}\phi(x,\xi,y)\,\coloneqq\,\frac{1}{2}\operatorname*{trace}\left(a(x)\nabla^{2}\phi(x)\right)+\bigl{\langle}b(x,\xi)+a(x)y,\nabla\phi(x)\bigr{\rangle}\,,$$

$$L(x,\xi ,y):=c(x,\xi )-\frac{1}{2}|{\text{upsigma}}^{\mathsf{T}}(x)y{|}^2,(x,\xi ,y)\in \text{mathds}{R}^d\times \mathcal{K}\times \text{mathds}{R}^d.$$

$${\mathcal{G}}f(x)\,\coloneqq\,\frac{1}{2}\operatorname*{trace}\left(a(x)\nabla^{2}f(x)\right)+\max_{\xi\in{\mathscr{K}}}\,\bigl{[}\bigl{\langle}b(x,\xi),\nabla f(x)\bigr{\rangle}+c(x,\xi)f(x)\bigr{]}$$

$${\mathcal{H}}(x)\,\coloneqq\,\frac{1}{2}\,\bigl{\lvert}\upsigma^{\mathsf{T}}(x)\nabla{\varphi_{\mspace{-2.0mu}*}}(x)\bigr{\rvert}^{2}\,,\quad x\in{\mathds{R}^{d}}\,,$$

$${\mathcal{M}}_{{\mathscr{A}}}\,\coloneqq\,\biggl{\{}\mu\in{\mathcal{P}}({\mathcal{Z}})\,\colon\int_{{\mathcal{Z}}}{\mathscr{A}}f(z)\,\mu(\mathrm{d}{z})\,=\,0\quad\forall\,f\in{\mathcal{C}}^{2}_{c}({\mathds{R}^{d}})\biggr{\}}\,,$$

$${\mathcal{P}}_{\text{mspace}-3.0mu\ast }(\mathcal{Z})$$

$${\mathcal{P}}_{\text{mspace}-3.0mu\circ }(\mathcal{Z})$$

$$J_{\ast }={\lambda }_{\ast }$$

$$=\underset{\mu \in {\mathcal{M}}_{\mathcal{A}}\cap {\mathcal{P}}_{\text{mspace}-3.0mu\ast }(\mathcal{Z})}{\max }{\int }_{\mathcal{Z}}L(z)\mu (\mathrm{d}z).$$

$$J_{*}\,=\,\adjustlimits{\inf}_{g\in{\mathcal{C}}^{2}_{c}({\mathds{R}^{d}})}{\sup}_{\mu\in{\mathcal{P}}({\mathcal{Z}})}\,\int_{{\mathcal{Z}}}\bigl{(}{\mathscr{A}}g(z)+{L}(z)\bigr{)}\,\mu(\mathrm{d}{z})\,.$$

$$\text{uptau}(A):=\inf \{t>0:X_t\notin A\}.$$

$$\mathrm{d}{X}_t=b(X_t,{\xi }_t)\mathrm{d}t+\text{upsigma}(X_t)\mathrm{d}{W}_t-\gamma (X_t)\mathrm{d}{\eta }_t,$$

$${\mathfrak{F}}_s:=\text{the completion of }\sigma \{X_0,{\xi }_r,W_r,r\le s\}\text{ relative to }(\mathfrak{F},P).$$

$${\gamma }_i(x)=\sum _{j=1}^d{a}^{ij}(x)n_j(x),x\in \partial Q,$$

$$J^{x}_{\xi}(c;Q)\,=\,\liminf_{T\to\infty}\,\frac{1}{T}\,\log\operatorname{\mathbb{E}}^{x}_{\xi}\Bigl{[}\mathrm{e}^{\int^{T}_{0}c(X_{t},\xi_{t})\,\mathrm{d}t}\Bigr{]}\,,\quad x\in Q\,,$$

$$J_{\ast }^x(c;Q):=\underset{\xi \in \Xi }{\sup }{J}_{\xi }^x(c;Q),x\in Q,\text{and }{J}_{\ast }(c;Q):=\underset{x\in Q}{\sup }{J}_{\ast }^x(c;Q).$$

$${\mathcal{C}}^{2}_{\gamma}(\overline{Q})\,\coloneqq\,\bigl{\{}f\in{\mathcal{C}}^{2}(\overline{Q})\,\colon\,\langle\nabla f,\gamma\rangle\,=\,0\text{\ on\ }\partial{Q}\bigr{\}}\,,$$

$${\mathcal{L}}_{\xi }f(x)$$

$$\mathcal{G}f(x)$$

$$S_{t}f(x)\,\coloneqq\,\sup_{\xi\in{\Xi}}\,\operatorname{\mathbb{E}}^{x}_{\xi}\Bigl{[}e^{\int_{0}^{t}c(X_{s},\xi_{s})\,\mathrm{d}{s}}f(X_{t})\Bigr{]}\,.$$

$$\mathcal{G}V=\rho V\text{ in }Q,⟨\nabla V,\gamma ⟩=0\text{ on }\partial Q,\text{and }V(0)=1.$$

$$J_{\ast }^x(c;Q)=J_{\ast }(c;Q)=\rho \forall x\in Q,$$

$$\rho =\text{adjustlimits}{\inf }_{f\in {\mathcal{C}}_{\gamma ,+}^2(\overline{Q}),f>0}{\sup }_{x\in \overline{Q}}\frac{\mathcal{G}f(x)}{f(x)}=\text{adjustlimits}{\sup }_{f\in {\mathcal{C}}_{\gamma ,+}^2(\overline{Q}),f>0}{\inf }_{x\in \overline{Q}}\frac{\mathcal{G}f(x)}{f(x)}.$$

$$L(x,\xi ,y):=c(x,\xi )-\frac{1}{2}|{\text{upsigma}}^{\mathsf{T}}(x)y{|}^2,(x,\xi ,y)\in \overline{Q}\times \mathcal{K}\times \text{mathds}{R}^d,$$

$${\mathscr{A}}\phi(x,\xi,y)\,\coloneqq\,\frac{1}{2}\operatorname*{trace}\left(a(x)\nabla^{2}\phi(x)\right)+\bigl{\langle}b(x,\xi)+a(x)y,\nabla\phi(x)\bigr{\rangle}\,.$$

$$\frac{{\mathcal{G}}f(x)}{f(x)}\,=\,\adjustlimits{\max}_{\xi\in{\mathscr{K}}}{\max}_{y\in{\mathds{R}^{d}}}\;\bigl{[}{\mathscr{A}}g(x,\xi,y)+{L}(x,\xi,y)\bigr{]}\,.$$

$$\rho$$

$$\displaystyle\,=\,\adjustlimits{\sup}_{g\in{\mathcal{C}}^{2}_{\gamma}(\overline{Q})\,}{\inf}_{x\in\overline{Q}\;}\sup_{\xi\in{\mathscr{K}},\,y\in{\mathds{R}^{d}}}\,\Bigl{(}{\mathscr{A}}g(x,\xi,y)+{L}(x,\xi,y)\Bigr{)}\,.$$

$$F(g,\mu)\,\coloneqq\,\int_{\overline{Q}\times{\mathscr{K}}\times{\mathds{R}^{d}}}\bigl{(}{\mathscr{A}}g(x,\xi,y)+{L}(x,\xi,y)\bigr{)}\,\mu(\mathrm{d}{x},\mathrm{d}{\xi},\mathrm{d}{y})$$

$$\rho =\text{adjustlimits}{\inf }_{g\in {\mathcal{C}}_{\gamma }^2(\overline{Q})}{\sup }_{\mu \in \mathcal{P}(\overline{Q}\times \mathcal{K}\times \text{mathds}{R}^d)}F(g,\mu ).$$

Full text

{AMS}

60J60, Secondary 60J25, 35K59, 35P15, 60F10

1 Introduction

In this paper we consider the risk-sensitive reward maximization problem on ${\mathds{R}^{d}}$ for diffusions controlled through the drift. The main objective is to derive a variational formulation for the risk-sensitive reward in the spirit of [2], which does so for discrete time problems on a compact state space, and analyze the associated Hamilton–Jacobi–Bellman (HJB) equation. Since the seminal work of Donsker and Varadhan [18, 19], this problem has acquired prominence. The variational formula derived here can be viewed as a controlled version of the variational formulas for principal eigenvalues of diffusion operators arising in large deviations. For reversible diffusions, this formula can be viewed as an abstract Courant–Fischer formula [18]. For general diffusions, the correct counterpart in linear algebra is the Collatz–Wielandt formula for the principal eigenvalue of non-negative matrices [27, Chapter 8]. For its connection with the large deviations theory for finite Markov chains and an equivalent variational description, see [17].

There has been considerable interest to generalize this theory to a natural class of nonlinear self-maps on positive cones of finite or infinite dimensional spaces. The first task is to establish the existence and where possible, uniqueness of the principal eigenvalue and eigenvector (the latter modulo a scalar multiple as usual), that is, a nonlinear variant of the Perron–Frobenius theorem in the finite dimensional case and its generalization, the Krein–Rutman theorem, in Banach spaces. This theory is carried out in, e.g., [25, 29]. The next problem is to derive an abstract Collatz–Wielandt formula for the principal eigenvalue [1]. In bounded domains, a Collatz–Wielandt formula for the Dirichlet principal eigenvalue of a convex nonlinear operator is obtained in [10]. Our first objective coincides with this, albeit for Feynman–Kac operators arising in risk-sensitive control that we introduce later. For risk-sensitive reward processes, that is, the problem of maximizing the asymptotic growth rate for the risk-sensitive reward in discrete time problems, one can go a step further and give an explicit characterization of the principal eigenvalue as the solution of a concave maximization problem [2]. The objective of this article is to carry out this program for controlled diffusions.

At this juncture, it is worthwhile to underscore the difference between reward maximization and cost minimization problems with risk-sensitive criteria. Unlike the more classical criteria such as ergodic or discounted, they cannot be converted from one to the other by a sign flip. The cost minimization criterion, after a logarithmic transformation applied to its HJB equation, leads to the Isaacs equation for a zero-sum stochastic differential game [20]. An identical procedure applied to the reward maximization problem would lead to a team problem wherein the two agents seek to maximize the same payoff non-cooperatively. The latter in particular implies that their decisions at any time are conditionally independent given the state (more generally, the past history). Our approach leads to a concave maximization problem, an immense improvement with potential implications for possible numerical schemes. This does not seem possible for the cost minimization problem. Thus the complexity of the latter is much higher. Recently, a risk-sensitive maximization problem is also studied in [14] under a blanket geometric stability condition. In the present paper we do not impose any blanket stability on the controlled processes.

We first establish these results for reflected diffusions in a bounded domain, for which the nonlinear Krein–Rutman theorem of [29] paves the way. This is not so if the state space is all of ${\mathds{R}^{d}}$. Extension to the whole space turns out to be quite involved due to the lack of compactness. Even the well-posedness of the underlying nonlinear eigenvalue problem is pretty tricky. Hence we proceed via the infinite volume limit of the finite volume problems. This leads to an abstract Collatz–Wielandt formula and an abstract Donsker–Varadhan formula. More specifically, in Theorem 3.4 we show that the generalized eigenvalue of the semilinear operator is simple, and identify some useful properties of its eigenvector. We proceed to prove equality between the risk-sensitive value and the generalized principal eigenvalue in Theorem 3.8, which also establishes a verification of optimality criterion. The general result for the variational formula is in Proposition 4.1, followed by more specialized results in Theorems 4.11 and 4.15. In the process of deriving these results, we present some techniques that may have wider applicability. Most prominent of these is perhaps the gradient estimate in Lemma 4.5 for operators with measurable coefficients.

Lastly, in Section 5 we revisit the risk-sensitive minimization problem, and with the aid of Lemma 4.5 we improve the main result in [3] by extending it to unbounded drifts and running costs, under suitable growth conditions (see Section 5).

1.1 A brief summary of the main results

We summarize here the results concerning the variational formula on the whole space. We consider a controlled diffusion in ${\mathds{R}^{d}}$ of the form

[TABLE]

defined in a complete probability space $(\Omega,{\mathfrak{F}},\operatorname{\mathbb{P}})$. The process $W$ is a $d$-dimensional standard Wiener process independent of the initial condition $X_{0}$, and the control process $\{\xi_{t}\}_{t\geq 0}$ lives in a compact metrizable space ${\mathscr{K}}$. We impose a standard set of assumptions on the coefficients which guarantee existence and uniqueness of strong solutions under all admissible controls. Namely, local Lipschitz continuity in $x$ and at most affine growth of $b$ and $\upsigma$, and local non-degeneracy of $a\coloneqq\upsigma\upsigma^{\mathsf{T}}$ (see Section 3 (i)). But we do not impose any ergodicity assumptions on the controlled diffusion. The process $\{X_{t}\}_{t\geq 0}$ could be transient.

We let $c\colon{\mathds{R}^{d}}\times{\mathscr{K}}\to\mathds{R}$ be a continuous running reward function, which is assumed bounded from above, and define the optimal risk-sensitive value $J_{*}$ by

[TABLE]

where the supremum is over all admissible controls, and $\operatorname{\mathbb{E}}$ denotes the expectation operator. This problem is translated to an ergodic control problem for the operator ${\mathscr{A}}\colon{\mathcal{C}}^{2}({\mathds{R}^{d}})\to{\mathcal{C}}({\mathds{R}^{d}}\times{\mathscr{K}}\times{\mathds{R}^{d}})$, defined by

[TABLE]

where $\nabla^{2}$ denotes the Hessian, and $a(x)=\upsigma(x)\upsigma^{\mathsf{T}}(x)$, that seeks to maximize the average value of the functional

[TABLE]

We first show that the generalized principal eigenvalue $\lambda_{*}$ (see Eq. 37) of the maximal operator

[TABLE]

is simple. An important hypothesis for this is that $c-\lambda_{*}$ is negative and bounded from above away from zero on the complement of some compact set (see Section 3 (iii)). This is always satisfied if $-c$ is an inf-compact function (i.e., the sublevel sets $\{-c\leq\kappa\}$ are compact, or empty, in ${\mathds{R}^{d}}\times{\mathscr{K}}$ for each $\kappa\in\mathds{R}$), or if $c$ is a positive function vanishing at infinity and the process $\{X_{t}\}_{t\geq 0}$ is recurrent under some stationary Markov control. Let the positive function $\Phi_{\mspace{-2.0mu}*}\in{\mathcal{C}}^{2}({\mathds{R}^{d}})$, normalized as $\Phi_{\mspace{-2.0mu}*}(0)=1$ to render it unique, denote the principal eigenvector, that is, ${\mathcal{G}}\Phi_{\mspace{-2.0mu}*}=\lambda_{*}\Phi_{\mspace{-2.0mu}*}$, and define ${\varphi_{\mspace{-2.0mu}*}}=\log\Phi_{\mspace{-2.0mu}*}$. The function

[TABLE]

plays a very important role in the analysis, and can be interpreted as an infinitesimal relative entropy rate (see Section 4). To keep the notation simple, we define ${\mathcal{Z}}\coloneqq{\mathds{R}^{d}}\times{\mathscr{K}}\times{\mathds{R}^{d}}$, and use the single variable $z=(x,\xi,y)\in{\mathcal{Z}}$. Let ${\mathcal{P}}({\mathcal{Z}})$ denote the set of probability measures on the Borel $\sigma$-algebra of ${\mathcal{Z}}$, and ${\mathcal{M}}_{A}$ denote the set of infinitesimal ergodic occupation measures for the operator ${\mathscr{A}}$ defined by

[TABLE]

where ${\mathcal{C}}^{2}_{c}({\mathds{R}^{d}})$ is the class of functions in ${\mathcal{C}}^{2}({\mathds{R}^{d}})$ which have compact support. We also define

[TABLE]

Then, under the mild hypotheses of Section 3, we show in Proposition 4.1 that

[TABLE]

We next specialize the results to the case where the diffusion matrix $a$ is bounded and uniformly elliptic (see Section 4), and show in Theorem 4.11 that under any of the hypotheses of Section 4 we have ${\mathcal{M}}_{{\mathscr{A}}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}\circ}}({\mathcal{Z}})\subset{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$. This permits us to replace ${{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$ with ${\mathcal{P}}({\mathcal{Z}})$ and ${\mathcal{M}}_{\mathscr{A}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$ with ${\mathcal{M}}_{\mathscr{A}}$ in the second and third equalities of Eq. 7, respectively. We note here that if $a$ is bounded and uniformly elliptic, then Section 4 is satisfied when either $-c$ is inf-compact, or $\langle b,x\rangle^{-}$ has subquadratic growth, or $\frac{\lvert b\rvert^{2}}{1+\lvert c\rvert}$ is bounded.

We also show that if $\frac{{\mathcal{H}}}{1+\lvert{\varphi_{\mspace{-2.0mu}*}}\rvert}$ is bounded (see Lemma 4.13 for explicit conditions on the parameters under which this holds), then we can commute the ‘$\sup$’ and the ‘$\inf$’ to obtain

[TABLE]

Also, in Theorem 4.15, we establish the variational formula over the class of functions in ${\mathcal{C}}^{2}({\mathds{R}^{d}})$ whose partial derivatives up to second order have at most polynomial growth in $\lvert x\rvert$.

1.2 Notation

The standard Euclidean norm in $\mathds{R}^{d}$ is denoted by $\lvert\,\cdot\,\rvert$, and $\mathds{N}$ stands for the set of natural numbers. The closure, the boundary and the complement of a set $A\subset\mathds{R}^{d}$ are denoted by $\bar{A}$, $\partial{A}$ and $A^{c}$, respectively. We denote by $\uptau(A)$ the first exit time of the process $\{X_{t}\}$ from the set $A\subset\mathds{R}^{d}$, defined by

[TABLE]

The open ball of radius $r$ in $\mathds{R}^{d}$, centered at $x\in{\mathds{R}^{d}}$, is denoted by $B_{r}(x)$, and $B_{r}$ is the ball centered at [math]. We let $\uptau_{r}\coloneqq\uptau(B_{r})$, and ${\breve{\uptau}}_{r}\coloneqq\uptau(B^{c}_{r})$. For a Borel space $Y$, ${\mathcal{P}}(Y)$ denotes the set of probability measures on its Borel $\sigma$-algebra.

The term domain in $\mathds{R}^{d}$ refers to a nonempty, connected open subset of the Euclidean space $\mathds{R}^{d}$. For a domain $D\subset\mathds{R}^{d}$, the space ${\mathcal{C}}^{k}(D)$ (${\mathcal{C}}^{k}_{b}(D)$) refers to the class of all real-valued functions on $D$ whose partial derivatives up to order $k$ exist and are continuous (and bounded). In addition ${\mathcal{C}}_{c}^{k}(D)$ denotes the class of functions in ${\mathcal{C}}^{k}(D)$ that have compact support. The space ${L}^{p}(D)$, $p\in[1,\infty)$, stands for the Banach space of (equivalence classes of) measurable functions $f$ satisfying $\int_{D}\lvert f(x)\rvert^{p}\,\mathrm{d}{x}<\infty$, and ${L}^{\infty}(D)$ is the Banach space of functions that are essentially bounded in $D$. The standard Sobolev space of functions on $D$ whose generalized derivatives up to order $k$ are in ${L}^{p}(D)$, equipped with its natural norm, is denoted by ${\mathscr{W}}^{k,p}(D)$, $k\geq 0$, $p\geq 1$.

In general, if $\mathcal{X}$ is a space of real-valued functions on $Q$, $\mathcal{X}_{\mathrm{loc}}$ consists of all functions $f$ such that $f\varphi\in\mathcal{X}$ for every $\varphi\in{\mathcal{C}}_{c}^{\infty}(Q)$, the space of smooth functions on $Q$ with compact support. In this manner we obtain for example the space ${\mathscr{W}}_{\text{loc}}^{2,p}(Q)$.

We adopt the notation $\partial_{t}\coloneqq\tfrac{\partial}{\partial{t}}$, and for $i,j\in\mathds{N}$, $\partial_{i}\coloneqq\tfrac{\partial~{}}{\partial{x}_{i}}$ and $\partial_{ij}\coloneqq\tfrac{\partial^{2}~{}}{\partial{x}_{i}\partial{x}_{j}}$, and use the standard summation rule that repeated subscripts and superscripts are summed from $1$ through $d$.

2 The problem on a bounded domain

In this section, we consider the risk-sensitive reward maximization with state dynamics given by a reflected diffusion on a bounded ${\mathcal{C}}^{2}$ domain $Q\subset{\mathds{R}^{d}}$ with co-normal direction of reflection. In particular, the dynamics are given by

[TABLE]

where $\eta_{t}$ denotes the local time of the process $X$ on the boundary $\partial Q$. The random processes in Eq. 8 live in a complete probability space $(\Omega,{\mathfrak{F}},\operatorname{\mathbb{P}})$. The process $W=(W_{t})_{t\geq 0}$ is a $d$-dimensional standard Wiener process independent of the initial condition $X_{0}$. The control process $\xi=(\xi_{t})_{t\geq 0}$ takes values in a compact, metrizable set ${\mathscr{K}}$, and $\xi_{t}(\omega)$ is jointly measurable in $(t,\omega)\in[0,\infty)\times\Omega$. The set of admissible controls ${\Xi}$ consists of the control processes $\xi$ that are non-anticipative: for $s<t$, $W_{t}-W_{s}$ is independent of

[TABLE]

Concerning the coefficients of the equation, we assume the following:

1. (i)

The drift $b$ is a continuous map from $\overline{Q}\times{\mathscr{K}}$ to ${\mathds{R}^{d}}$, and Lipschitz in its first argument uniformly with respect to the second. 2. (ii)

The diffusion matrix $\upsigma\colon\overline{Q}\to\mathds{R}^{d\times d}$ is continuously differentiable, its derivatives are Hölder continuous, and is non-degenerate in the sense that the minimum eigenvalue of a(x)=\bigl{[}a^{ij}(x)\bigr{]}\coloneqq\upsigma(x)\upsigma^{\mathsf{T}}(x) on $Q$ is bounded away from zero. 3. (iii)

The reflection direction $\gamma=[\gamma_{1}(x),\dotsc,\gamma_{d}(x)]^{\mathsf{T}}\colon{\mathds{R}^{d}}\to{\mathds{R}^{d}}$ is co-normal, that is, $\gamma$ is given by

[TABLE]

where $\vec{n}(x)=[n_{1}(x),\dotsc,n_{d}(x)]^{\mathsf{T}}$ is the unit outward normal.

We let ${\Xi_{\mathsf{sm}}}$ denote the set of stationary Markov controls, that is, the set of Borel measurable functions $v\colon{\mathds{R}^{d}}\to{\mathscr{K}}$. Given $\xi\in{\Xi}$, the stochastic differential equation in Eq. 8 has a unique strong solution. The same is true for the class of Markov controls [8, Chapter 2]. Let $\operatorname{\mathbb{P}}^{x}_{\xi}$ and $\operatorname{\mathbb{E}}^{x}_{\xi}$ denote the probability measure and expectation operator on the canonical space of the process controlled under $\xi\in{\Xi}$, with initial condition $X_{0}=x$.

Given a continuous reward function $c\colon\overline{Q}\times{\mathscr{K}}\to\mathds{R}$, which is Lipschitz continuous in its first argument uniformly with respect to the second, the objective of the risk-sensitive reward problem is to maximize

[TABLE]

over all admissible controls $\xi\in{\Xi}$. We define

[TABLE]

The solution of this problem shows that $J^{x}_{*}(c;Q)$ does not depend on $x$.

We let

[TABLE]

and ${\mathcal{C}}^{2}_{\gamma,+}(\overline{Q})$ denote its subspace consisting of nonnegative functions.

For $f\in{\mathcal{C}}^{2}(\overline{Q})$, and $\xi\in{\mathscr{K}}$, we define

[TABLE]

We summarize some results from [9] that are needed in Theorem 2.1 below. Without loss of generality we assume that $0\in Q$.

Consider the operator $S_{t}\colon{\mathcal{C}}(\overline{Q})\to{\mathcal{C}}(\overline{Q})$, $t\in\mathds{R}_{+}$, defined by

[TABLE]

The characterization of $S_{t}$ is exactly analogous to [9, Theorem 3.2], which considers the minimization problem (see also [9, Remark 4.2]). Specifically, for each $f\in C^{2+\delta}_{\gamma}(\overline{Q})$, and $T>0$, the quasi-linear parabolic p.d.e. $\partial_{t}\,u(t,x)={\mathcal{G}}u(t,x)$ in $(0,T]\times Q$, with $u(0,x)=f(x)$ for all $x\in\overline{Q}$, and $\langle\nabla u(t,x),\gamma(x)\rangle=0$ for all $(t,x)\in(0,T]\times\partial{Q}$, has a unique solution in {\mathcal{C}}^{1+\nicefrac{{\delta}}{{2}},2+\delta}\bigl{(}[0,T]\times\overline{Q}\bigr{)}. This solution has the stochastic representation $u(t,x)\,=\,S_{t}f(x)$ for all $(t,x)\in[0,T]\times\overline{Q}$.

Following the analysis in [9] we obtain the following characterization of $J_{*}(c;Q)$ defined in Eq. 11.

Theorem 2.1**.**

There exists a unique pair $(\rho,V)\in\mathds{R}\times{\mathcal{C}}^{2}_{\gamma,+}(\overline{Q})$ which solves

[TABLE]

Also, $S_{t}V(x)=e^{\rho t}V(x)$, for $(x,t)\in\overline{Q}\times[0,\infty)$. In addition, we have

[TABLE]

and

[TABLE]

Proof 2.2**.**

Equation 14* is the result in [9, Lemma 2.1], while the other assertions follow from Lemma 4.5 and Remark 4.2 in [9]. *

2.1 A variational formula

Define

[TABLE]

and an operator ${\mathscr{A}}\colon{\mathcal{C}}^{2}_{\gamma}(\overline{Q})\to{\mathcal{C}}({\mathds{R}^{d}}\times{\mathscr{K}}\times{\mathds{R}^{d}})$ by

[TABLE]

It is important to note that if $f\in{\mathcal{C}}^{2}_{\gamma,+}(\overline{Q})$ is a positive function and $g=\log f$, then

[TABLE]

Thus, we obtain from Eq. 14 that

[TABLE]

We let

[TABLE]

for $g\in{\mathcal{C}}^{2}_{\gamma}(\overline{Q})$ and $\mu\in{\mathcal{P}}(\overline{Q}\times{\mathscr{K}}\times{\mathds{R}^{d}})$.

It is clear that Eq. 15 can be written as

[TABLE]

Let ${\mathcal{M}}_{{\mathscr{A}},Q}$ denote the class of infinitesimal ergodic occupation measures for the operator ${\mathscr{A}}$, defined by

[TABLE]

Implicit in this definition is the requirement that $\int\lvert{\mathscr{A}}f\rvert\,\mathrm{d}\mu<\infty$ for all $f\in{\mathcal{C}}^{2}_{\gamma}(\overline{Q})$ and $\mu\in{\mathcal{M}}_{{\mathscr{A}},Q}$. We have the following result.

Theorem 2.3**.**

It holds that

[TABLE]

Moreover, ${\mathcal{P}}(\overline{Q}\times{\mathscr{K}}\times{\mathds{R}^{d}})$ may be replaced with ${\mathcal{M}}_{{\mathscr{A}},Q}$ in Eq. 18, and thus

[TABLE]

Proof 2.4**.**

The first equality in Eq. 18 follows by Eq. 17. We continue to prove the rest of the assertions. First note that

[TABLE]

because the infimum on the left hand side is $-\infty$ for $\mu\notin{\mathcal{M}}_{{\mathscr{A}},Q}$. It follows by Eq. 17 that $\hat{\rho}\leq\rho$. Let $v_{*}$ be a measurable selector from the maximizer of Eq. 13, that is,

[TABLE]

With $\phi\coloneqq\log V$, Eq. 13 takes the form

[TABLE]

The reflected diffusion with drift b\bigl{(}x,v_{*}(x)\bigr{)}+a(x)\nabla\phi(x) is of course exponentially ergodic. Let $\eta_{*}$ denote its invariant probability measure. Then, Eq. 19 implies that

[TABLE]

Let $\mu_{*}\in{\mathcal{P}}(\overline{Q}\times{\mathscr{K}}\times{\mathds{R}^{d}})$ be defined by

[TABLE]

where $\delta_{y}$ denotes the Dirac mass at $y$. Then $\mu_{*}$ is an ergodic occupation measure for the controlled reflected diffusion with drift $b(x,\xi)+a(x)y$, and thus $\mu_{*}\in{\mathcal{M}}_{{\mathscr{A}},Q}$. Let $g\in{\mathcal{C}}^{2}_{\gamma}(\overline{Q})$ be arbitrary. Then

[TABLE]

*where the second equality follows by Eq. 20. Thus $\hat{\rho}\geq\rho$, and since we have already asserted the reverse inequality, we must have equality. This establishes Eq. 18, and also proves the last assertion of the theorem. *

3 The risk-sensitive reward problem on ${\mathds{R}^{d}}$

In this section we study the risk-sensitive reward maximization problem on ${\mathds{R}^{d}}$. We consider a controlled diffusion of the form

[TABLE]

All random processes in Eq. 21 live in a complete probability space $(\Omega,{\mathfrak{F}},\operatorname{\mathbb{P}})$. The control process $\{\xi_{t}\}_{t\geq 0}$ lives in a compact metrizable space ${\mathscr{K}}$.

We approach the problem in ${\mathds{R}^{d}}$ as a limit of Dirichlet or Neumann eigenvalue problems on balls $B_{r}$, $r>0$. Differentiability of the matrix $a$ can be relaxed here. Consider the eigenvalue problem on a ball $B_{r}$, with Neumann boundary conditions, and the reflection direction along the exterior normal $\vec{n}(x)$ to $B_{r}$ at $x$. The drift $b:\bar{B}_{r}\times{\mathscr{K}}\to{\mathds{R}^{d}}$ is continuous, and Lipschitz in its first argument uniformly with respect to the second. The diffusion matrix $a$ is Lipschitz continuous on $\bar{B}_{r}$ and non-degenerate. Let $\rho_{r}$ denote the principal eigenvalue on $B_{r}$ under Neumann boundary conditions of the operator ${\mathcal{G}}$ defined in Eq. 12. We refer to $\rho_{r}$ as the Neumann eigenvalue on $B_{r}$. It follows from the results in [30] (see in particular Theorems 5.1, 6.6, and Proposition 7.1) that there exists a unique $V_{r}\in{\mathcal{C}}^{2}(B_{r})\cap{\mathcal{C}}^{0,1}(\bar{B}_{r})$, with $V_{r}>0$ on $B_{r}$ and $V_{r}(0)=1$, solving

[TABLE]

and $\langle\nabla V_{r}(x),\vec{n}(x)\rangle=0$ on $\partial B_{r}$. We also refer the reader to [24, Theorem 12.1, p. 195].

We adopt the following structural hypotheses on the coefficients of Eq. 21 and the reward function $c$ have the following structural properties.

{assumption}

1. (i)

The drift $b\colon\mathds{R}^{d}\times{\mathscr{K}}\to\mathds{R}^{d}$ is continuous, and for some constant $C_{R}>0$ depending on $R>0$, we have

[TABLE]

where \lVert\upsigma\rVert\coloneqq\bigl{(}\operatorname*{trace}\,\upsigma\upsigma^{\mathsf{T}}\bigr{)}^{\nicefrac{{1}}{{2}}} denotes the Hilbert–Schmidt norm of $\upsigma$. 2. (ii)

The reward function $c\colon{\mathds{R}^{d}}\times{\mathscr{K}}\to\mathds{R}$ is continuous and locally Lipschitz in its first argument uniformly with respect to $\xi\in{\mathscr{K}}$, is bounded from above in ${\mathds{R}^{d}}$, and $x\mapsto\max_{\xi\in{\Xi}}\,\lvert c(x,\xi)\rvert$ has polynomial growth in $\lvert x\rvert$. 3. (iii)

We assume that the Neumann eigenvalues $\rho_{n}$ satisfy

[TABLE]

Section 3 is enforced throughout the rest of the paper, unless mentioned otherwise. Part (i) of this assumption are the usual hypotheses that guarantee existence and uniqueness of strong solutions to Eq. 21 under any admissible control.

Remark 3.1**.**

Equation 24* is a version of the near-monotone assumption, which is often used in ergodic control problems (see [8]). This has the effect of penalizing instability, ensuring tightness of laws for optimal controls. There are two important cases where Eq. 24 is always satisfied. First, when $-c$ is inf-compact. In this case we have ${\rho_{*}}\leq\sup_{{\mathds{R}^{d}}\times{\mathscr{K}}}c$ and ${\rho_{*}}>-\infty$, since the Dirichlet eigenvalues which are a lower bound for ${\rho_{*}}$ are increasing as a function of the domain [7, Lemma 2.1]. Second, when $c$ is positive and vanishes at infinity, and under some stationary Markov control the process $\{X_{t}\}_{t\geq 0}$ in Eq. 21 is recurrent. This can be established by comparing $\rho_{n}$ with the Dirichlet eigenvalue on $B_{n}$ (see Section 3.2), and using [7, Theorems 2.6 and 2.7 (ii)]. For related studies concerning the class of running reward functions vanishing at infinity, albeit in the uncontrolled case, see [22, 23, 7, 10]. See also [4, Theorem 2.12] which studies the Collatz–Wielandt formula for the risk-sensitive minimization problem. *

Recall that ${\Xi_{\mathsf{sm}}}$ denotes the set of stationary Markov controls. For $v\in{\Xi_{\mathsf{sm}}}$, we use the simplifying notation

[TABLE]

and define ${\mathcal{L}}_{v}$ analogously.

We next review some properties of eigenvalues of linear and semilinear operators on ${\mathds{R}^{d}}$. For $f\in{\mathcal{C}}^{2}({\mathds{R}^{d}})$ and $\psi\in{\mathscr{W}}_{\text{loc}}^{2,d}({\mathds{R}^{d}})$, define

[TABLE]

with ${\mathcal{L}}_{\xi}$ as in Eq. 12. Let $v\in{\Xi_{\mathsf{sm}}}$. Suppose that a positive function $\Psi\in{\mathscr{W}}_{\text{loc}}^{2,d}({\mathds{R}^{d}})$ and $\lambda\in\mathds{R}$ solve the equation

[TABLE]

We refer to any such solution $(\Psi,\lambda)$ as an eigenpair of the operator ${\mathcal{L}}_{v}+c_{v}$, and we say that $\Psi$ is an eigenvector with eigenvalue $\lambda$. Note that by eigenvector we always mean a positive function. Let $\psi=\log\Psi$. We refer to the Itô stochastic differential equation

[TABLE]

as the twisted SDE, and to its solution as the twisted process corresponding to $\Psi$. Clearly $\widetilde{\mathcal{L}}^{\psi}_{v}$ is the extended generator of Eq. 27.

We define the generalized principal eigenvalue $\lambda_{v}=\lambda_{v}(c_{v})$ of the operator ${\mathcal{L}}_{v}+c_{v}$ by

[TABLE]

A principal eigenvector $\Psi_{\mspace{-2.0mu}v}\in{\mathscr{W}}_{\text{loc}}^{2,d}({\mathds{R}^{d}})$ is a positive solution of Eq. 26 with $\lambda=\lambda_{v}$. A principal eigenvector is also called a ground state, and we refer to the corresponding twisted SDE and twisted process as a ground state SDE and ground state process respectively. Unlike what is common in criticality theory, our definition of a ground state does not require the minimal growth property of the principal eigenfunction (see [6]).

An easy calculation shows that any eigenpair $(\Psi,\lambda)$ of ${\mathcal{L}}_{v}+c_{v}$ satisfies

[TABLE]

with $\psi=\log\Psi$. In other words, $(\Psi^{-1},-\lambda)$ is an eigenpair of $\widetilde{\mathcal{L}}^{\psi}_{v}-c_{v}$. Note also that $(\psi,\lambda)$ is a solution to the ‘linear’ eigenvalue equation

[TABLE]

and that this equation can also be written as

[TABLE]

An extensive study of generalized principal eigenvalues with applications to risk-sensitive control can be found in [3, 7]. In these papers, the ‘potential’ $c_{v}$ is assumed to be bounded below in ${\mathds{R}^{d}}$, so the results cannot be quoted directly. It is not our intention to reproduce all these results for potentials which are bounded above, so we only focus on results that are needed later in this paper. We only quote results in [3, 7] which do not depend on the assumption that $c_{v}$ is bounded below. Generally speaking, caution should be exercised with arguments in [3, 7] that employ the Fatou lemma. On the other hand, since $c$ usually appears in the exponent, invoking Fatou’s lemma hardly ever poses any problems.

Suppose that the twisted process in Eq. 27 is regular, that is, the solution exists for all times. Then, an application of [7, Lemma 2.3] shows that an eigenvector $\Psi$ has the stochastic representation (semigroup property)

[TABLE]

Recall that ${\breve{\uptau}}_{r}$ denotes the first hitting time of the ball $B_{r}$, for $r>0$. We need the following lemma.

Lemma 3.2**.**

We assume only Section 3 (i)–(ii). The following hold.

1. (a)

If $(\Psi,\lambda)$ is an eigenpair of ${\mathcal{L}}_{v}+c_{v}$ under some $v\in{\Xi_{\mathsf{sm}}}$, and the twisted process in Eq. 27 is exponentially ergodic, then we have the stochastic representation

[TABLE]

In addition, $\lambda=\lambda_{v}$, the generalized principal eigenvalue of ${\mathcal{L}}_{v}+c_{v}$, and the ground state $\Psi=\Psi_{\mspace{-2.0mu}v}$ is unique up to multiplication by a positive constant. 2. (b)

Any eigenpair $(\Psi,\lambda)\in{\mathscr{W}}_{\text{loc}}^{2,d}({\mathds{R}^{d}})\times{\mathds{R}^{d}}$ of ${\mathcal{L}}_{v}+c_{v}$ satisfying Eq. 32 is a principal eigenpair, and $\lambda$ is a simple eigenvalue.

Proof 3.3**.**

Combining the proof of [7, Theorem 2.2] with [7, Theorem 3.1], we deduce that for every $r>0$, there exists a $\delta>0$ such that

[TABLE]

Applying the Itô formula to Eq. 26 we obtain

[TABLE]

We study separately the three integrals on the right-hand side of Eq. 34, which we denote as $\mathscr{J}_{i}$, $i=1,2,3$. For the first integral we have

[TABLE]

by monotone convergence. Note that the limit is also finite by Eq. 33.

Let $\widetilde{\operatorname{\mathbb{P}}}^{x}_{\psi,v}$ and $\widetilde{\operatorname{\mathbb{E}}}^{x}_{\psi,v}$ denote the probability measure and expectation operator on the canonical space of the twisted process in Eq. 27 with initial condition $\tilde{X}_{0}=x$. Next, using again the technique in [7, Theorem 2.2], we write

[TABLE]

where in the second inequality we apply [7, Lemma 2.3]. Thus, $\mathscr{J}_{2}$ vanishes as $t\to\infty$.

Concerning $\mathscr{J}_{3}$, using monotone convergence, we obtain

[TABLE]

where the inequality follows from the proof of [7, Lemma 2.3]. In turn, the right-hand side of Eq. 35 vanishes as $n\to\infty$, since the twisted process is geometrically ergodic. This completes the proof of Eq. 32.

Suppose that a positive $\phi\in{\mathscr{W}}_{\text{loc}}^{2,d}({\mathds{R}^{d}})$ and $\hat{\lambda}\leq\lambda$ solve

[TABLE]

An application of Itô’s formula and Fatou’s lemma then shows that

[TABLE]

Equations 32* and 36 imply that if we scale $\phi$ by multiplying it with a positive constant until it touches $\Psi$ at one point from above, the function $\frac{\phi}{\Psi}$ attains its minimum value of $1$ at some point in $\bar{B}_{r}$. A standard calculation shows that*

[TABLE]

Thus, $\frac{\phi}{\Psi}$ must equal a constant by the strong maximum principle, which implies that $\hat{\lambda}=\lambda$. This of course means that $\lambda=\lambda_{v}$. Uniqueness of $\Psi_{\mspace{-2.0mu}v}$ is evident from the preceding argument. This completes the proof of part (a).

*Part (b) is evident from the preceding paragraph. This completes the proof. *

3.1 The Bellman equation in ${\mathds{R}^{d}}$

Recall the solution $(V_{r},\rho_{r})$ of (22), the definition of ${\rho_{*}}$ in Eq. 24, and the definition of ${\mathcal{G}}$ in Eq. 3. We define

[TABLE]

Recall the definitions of ${\mathscr{A}}$ and ${L}$ in Eqs. 1 and 2. Note that if $(\Phi,\lambda)$ is an eigenpair of ${\mathcal{G}}$, then similarly to Eq. 31, we have

[TABLE]

with $\varphi=\log\Phi$.

Theorem 3.4**.**

There exists $\Phi_{\mspace{-2.0mu}*}\in{\mathcal{C}}^{2}({\mathds{R}^{d}})$ satisfying

[TABLE]

and the following hold:

1. (a)

The function $\Phi_{\mspace{-2.0mu}*}^{-1}$ is inf-compact. 2. (b)

If $v_{*}$ is an a.e. measurable selector from the maximizer of Eq. 39, then, the diffusion with extended generator $\widetilde{\mathcal{L}}_{v_{*}}^{{\varphi_{\mspace{-2.0mu}*}}}$, as defined in Eq. 25, is exponentially ergodic and satisfies

[TABLE]

with ${\varphi_{\mspace{-2.0mu}*}}\coloneqq\log\Phi_{\mspace{-2.0mu}*}$. 3. (c)

${\rho_{*}}=\lambda_{*}$. 4. (d)

$\rho_{n}\to{\rho_{*}}$* and $V_{n}\to\Phi_{\mspace{-2.0mu}*}$ as $n\to\infty$ uniformly on compact sets, and the solution $\Phi_{\mspace{-2.0mu}*}$ to Eq. 39 is unique up to a scalar multiple, and satisfies*

[TABLE]

for all $r>0$, and for all $v\in{\Xi_{\mathsf{sm}}}$, with equality if and only if $v$ is an a.e. measurable selector from the maximizer in Eq. 39.

Proof 3.5**.**

Using Theorem 2.1 and (10)-(11), it follows that $\rho_{n}\leq\sup_{{\mathds{R}^{d}}\times{\mathscr{K}}}c$, and this combined with Section 3 (iii) shows that $\{\rho_{n}\}$ converges along some subsequence $\{n_{k}\}_{k\in\mathds{N}}\subset\mathds{N}$ to ${\rho_{*}}$. Therefore, the convergence of $V_{n_{k}}$ along some further subsequence $\{n_{k}^{\prime}\}\subset\{n_{k}\}$ to a $\Phi_{\mspace{-2.0mu}*}$ satisfying Eq. 39 follows as in the proof of [13, Lemma 2.1].

We now turn to part (a). Here in fact we show that $-\lvert{\varphi_{\mspace{-2.0mu}*}}\rvert$ has at least logarithmic growth in $\lvert x\rvert$. Let $\delta\in(0,1)$ be a constant such that ${\rho_{*}}-c(x,\xi)>4\delta$ for all $x$ outside some compact set in ${\mathds{R}^{d}}$. Consider a function of the form \phi(x)=\bigl{(}1+\lvert x\rvert^{2}\bigr{)}^{-\theta}, with $\theta>0$. By Item (i), there exists $\theta>0$ and $r_{\circ}>0$ such that

[TABLE]

We fix such a constant $\theta$. We restrict our attention to solutions $(V_{n},\rho_{n})$ of Eq. 22 over an increasing sequence in $\mathds{N}$, also denoted as $\{n\}$, such that $\rho_{n}$ converges to ${\rho_{*}}$. It is clear then that we may enlarge the radius $r_{\circ}$, if needed, so that

[TABLE]

Next, let $\breve{\chi}\colon\mathds{R}\to(0,\infty)$ be a convex function in ${\mathcal{C}}^{2}(\mathds{R})$ such that $\breve{\chi}(t)=t$ for $t\geq 2$, and $\breve{\chi}(t)$ is constant and positive for $t\leq 1$. This can be chosen so that $\breve{\chi}^{\prime\prime}<2$ and $\sup_{t>0}\,t\breve{\chi}^{\prime\prime}(t)<2$. Such a function can be constructed by requiring, for example, that $\breve{\chi}^{\prime\prime}(t)=6(2-t)(t-1)$ for $t\in[1,2]$, from which we obtain $\breve{\chi}(t)=-\frac{1}{2}t^{4}+3t^{3}-6t^{2}+5t$ for $t\in[1,2]$. A simple calculation shows that $\breve{\chi}(1)=\frac{3}{2}$. Note that $\breve{\chi}(t)-t\breve{\chi}^{\prime}(t)\geq 0$ for all $t>0$ by convexity. Let \breve{\chi}_{\epsilon}(t)\coloneqq\epsilon\breve{\chi}\bigl{(}\nicefrac{{t}}{{\epsilon}}\bigr{)} for $\epsilon>0$. Then

[TABLE]

Using Eqs. 42, 43, and 44, we obtain

[TABLE]

For the last inequality in Eq. 45, we use the properties $\breve{\chi}_{\epsilon}(\phi)\geq\phi\,\breve{\chi}_{\epsilon}^{\prime}(\phi)$ and $\phi\,\breve{\chi}_{\epsilon}^{\prime\prime}(\phi)<2$ from Eq. 44, that the fact that $\breve{\chi}_{\epsilon}(\phi)\geq\phi$ and $\delta<1$. Note that, due to radial symmetry, the support of \breve{\chi}^{\prime}_{\epsilon}\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}\phi is a ball of the form $B_{R_{\epsilon}}$, with $\epsilon\mapsto R_{\epsilon}$ an nonincreasing continuous function with $R_{\epsilon}\to\infty$ as $\epsilon\searrow 0$. Recall the functions $V_{n}$ in Eq. 22. Select $\epsilon$ such that $R_{\epsilon}=n>r_{\circ}$. Scale $V_{n}$ until it touches \breve{\chi}_{\epsilon}\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}\phi at some point $\hat{x}$ from below. Here, \breve{\chi}_{\epsilon}\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}\phi denotes the composition of $\breve{\chi}_{\epsilon}$ and $\phi$. Let $v_{n}$ be a measurable selector from the minimizer in Eq. 22, and define h_{n}\coloneqq\breve{\chi}_{\epsilon}\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}\phi-V_{n}. Then, by Eqs. 22 and 45, we have

[TABLE]

and $\langle\nabla h_{n},\gamma\rangle=0$ on $\partial B_{n}$, since the gradient of \breve{\chi}_{\epsilon}\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}\phi vanishes on $\partial B_{R_{\epsilon}}$. It follows by the strong maximum principle that $\hat{x}$ cannot lie in the $B_{n}\setminus B_{r_{\circ}}$. Thus $h_{n}>0$ on this set. This implies that $\hat{x}$ cannot lie on $\partial B_{n}$ either, without contradicting the Hopf boundary point lemma. Thus $\hat{x}\in B_{r_{\circ}}$. This however shows by taking limits as $\epsilon\searrow 0$, and employing the Harnack inequality which asserts that $V_{n}(x)\leq C_{\mathsf{H}}V_{n}(y)$ for all $x,y\in B_{r_{\circ}}$ for some constant $C_{\mathsf{H}}$, that $\Phi_{\mspace{-2.0mu}*}\leq C\phi$ for some constant $C$. This proves part (a).

Equation 40* follows by Eq. 29. Since $\Phi_{\mspace{-2.0mu}*}^{-1}$ is inf-compact and the right hand side of Eq. 40 is negative and bounded away from zero outside a compact set by Section 3 (iii), the associated diffusion is ergodic [22, Theorem 4.1]. In turn, the Foster–Lyapunov equation in Eq. 40 shows that the diffusion is exponentially ergodic [28]. This proves part (b).*

Moving to the proof of part (c), suppose that for some $\rho\leq{\rho_{*}}$ we have

[TABLE]

Evaluating this equation at measurable selector $v_{*}$ from the maximizer of Eq. 39, and following the argument in the proof of Lemma 3.2 we obtain $\rho={\rho_{*}}$ and $\phi=\Phi_{\mspace{-2.0mu}*}$. This also shows that ${\rho_{*}}\geq\lambda_{*}$ by the definition in Eq. 37, and thus we have equality by Eq. 39.

*In order to prove part (d), suppose that $\rho_{n}\to\rho\leq{\rho_{*}}$ along some subsequence. Taking limits along perhaps a further subsequence, we obtain a positive function $\phi\in{\mathcal{C}}^{2}({\mathds{R}^{d}})$ that satisfies Eq. 46 with equality. Thus $\rho={\rho_{*}}$ and and $\phi=\Phi_{\mspace{-2.0mu}*}$ by part (c). The stochastic representation in Eq. 41 follows as in the proof of Lemma 3.2. This completes the proof. *

3.2 Dirichlet eigenvalues and the risk-sensitive value

In this section we first show that the problem in ${\mathds{R}^{d}}$ can also be approached by using Dirichlet eigensolutions. The main result is Theorem 3.8, which establishes that ${\rho_{*}}$ equals the risk-sensitive value $J_{*}$, and the usual verification of optimality criterion.

We borrow some results from [11, 12]. These can also be found in [3, Lemma 2.2], and are summarized as follows: Fix any $v\in{\Xi_{\mathsf{sm}}}$. For each $r\in(0,\infty)$ there exists a unique pair (\Psi_{\mspace{-2.0mu}v,r},\lambda_{v,r})\in\bigl{(}{\mathscr{W}}^{2,p}(B_{r})\cap{\mathcal{C}}(\bar{B}_{r})\bigr{)}\times\mathds{R}, for any $p>d$, satisfying $\Psi_{\mspace{-2.0mu}v,r}>0$ on $B_{r}$, $\Psi_{\mspace{-2.0mu}v,r}=0$ on $\partial B_{r}$, and $\Psi_{\mspace{-2.0mu}v,r}(0)=1$, which solves

[TABLE]

Moreover, the solution has the following properties:

1. (i)

The map $r\mapsto\lambda_{v,r}$ is continuous and strictly increasing. 2. (ii)

In its dependence on the function $c_{v}$, $\lambda_{v,r}$ is nondecreasing, convex, and Lipschitz continuous (with respect to the ${L}^{\infty}$ norm) with Lipschitz constant $1$. In addition, if $c_{v}\lneqq c_{v}^{\prime}$ then $\lambda_{v,r}(c_{v})<\lambda_{v,r}(c_{v}^{\prime})$.

We refer to $\lambda_{v,r}$ and $\Psi_{\mspace{-2.0mu}v,r}$ as the (Dirichlet) eigenvalue and eigenfunction, respectively, of the operator ${\mathcal{L}}_{v}+c_{v}$ on $B_{r}$.

Recall the definition of ${\mathcal{G}}$ in Eq. 3. Based on the results in [31], there exists a unique pair (\Psi_{\mspace{-2.0mu}*,r},\lambda_{*,r})\in\bigl{(}{\mathcal{C}}^{2}(B_{r})\cap{\mathcal{C}}(\bar{B}_{r})\bigr{)}\times\mathds{R}, satisfying $\Psi_{\mspace{-2.0mu}*,r}>0$ on $B_{r}$, $\Psi_{\mspace{-2.0mu}*,r}=0$ on $\partial B_{r}$, and $\Psi_{\mspace{-2.0mu}*,r}(0)=1$, which solves

[TABLE]

and properties (i)–(ii) above hold for $\lambda_{*,r}$. Also recall the definitions of the generalized principal eigenvalues in Eqs. 28 and 37, and $\rho_{r}$ defined in Eq. 22.

Lemma 3.6**.**

The following hold:

1. (i)

For $r>0$, we have $\lambda_{v,r}\leq\lambda_{*,r}$ for all $v\in{\Xi_{\mathsf{sm}}}$, and $\lambda_{*,r}<\rho_{r}$. 2. (ii)

$\lim_{r\to\infty}\,\lambda_{v,r}=\lambda_{v}$* for all $v\in{\Xi_{\mathsf{sm}}}$, and $\lim_{r\to\infty}\,\lambda_{*,r}=\lambda_{*}$.*

Proof 3.7**.**

Part (i) is a straightforward application of the strong maximum principle. By Eqs. 12 and 48 we have

[TABLE]

Let $r^{\prime}<r$, and suppose that $\lambda_{v,r^{\prime}}\,\geq\,\lambda_{*,r}$. Scale $\Psi_{\mspace{-2.0mu}v,r^{\prime}}$ so that it touches $\Psi_{\mspace{-2.0mu}*,r}$ at one point from below in $B_{r^{\prime}}$. Then $\Psi_{\mspace{-2.0mu}*,r}-\Psi_{\mspace{-2.0mu}v,r^{\prime}}$ is nonnegative, and by Eqs. 47 and 49 it satisfies

[TABLE]

This however implies that $\Psi_{\mspace{-2.0mu}*,r}=\Psi_{\mspace{-2.0mu}v,r^{\prime}}$ on $B_{r^{\prime}}$ which is a contradiction. Hence $\lambda_{v,r^{\prime}}\,<\,\lambda_{*,r}$ for all $r^{\prime}<r$ and the inequality $\lambda_{v,r}\leq\lambda_{*,r}$ follows by the continuity of $r\mapsto\lambda_{v,r}$. Following the same method, with $r^{\prime}=r$, we obtain $\lambda_{*,r}<\rho_{r}$.

*Part (ii) follows by [7, Lemma 2.2 (ii)]. *

Recall the definitions in Eqs. 10 and 11, and let

[TABLE]

and similarly for $J^{x}_{*}$ and $J_{*}$. Also, recall that

[TABLE]

The theorem that follows concerns the equality $\lambda_{*}=J_{*}$. Recall the definition in Eq. 24.

Theorem 3.8**.**

*We have $\lambda_{*}={\rho_{*}}=J_{*}$. In addition, $J^{x}_{v}=J_{*}$ if and only if $v$ is an a.e. measurable selector from the maximizer of Eq. 39. *

Proof 3.9**.**

We already have ${\rho_{*}}=\lambda_{*}$ from Theorem 3.4. This also gives

[TABLE]

Choose $R>0$ such that ${\rho_{*}}>\sup_{B^{c}_{R}\times{\mathscr{K}}}\,c$. This is possible by Eq. 24. Let $\delta>0$ be given, and select a smooth, non-negative cut-off function $\chi$ that vanishes in $B_{R}$ and equals to $1$ in $B_{R+1}^{c}$. Let $\Psi=\Phi_{\mspace{-2.0mu}*}+\varepsilon\chi$, and select $\epsilon>0$ small enough so that

[TABLE]

This is clearly possible since $\Phi_{\mspace{-2.0mu}*}$ is positive and

[TABLE]

We have

[TABLE]

Since $\Psi$ is bounded below away from zero, a standard use of Itô’s formula and the Fatou lemma applied to Eq. 50 shows that $J^{x}_{\xi}\leq{\rho_{*}}+\delta$ for all $\xi\in{\Xi}$. Since $\delta$ is arbitrary this implies ${\rho_{*}}\geq J_{*}$, and hence we must have equality. This also shows that every a.e. measurable selector from the maximizer of Eq. 39 is optimal.

Next, for $v\in{\Xi_{\mathsf{sm}}}$, let $(\lambda_{v},\Psi_{\mspace{-2.0mu}v})$ be an eigenpair, obtained as a limit of Dirichlet eigenpairs \bigl{\{}(\lambda_{v,n},\Psi_{\mspace{-2.0mu}v,n})\bigr{\}}_{n\in\mathds{N}}, with $\Psi_{\mspace{-2.0mu}v,n}(0)=1$, along some subsequence (see Lemma 3.6). Let $\nu\in[-\infty,\infty)$ be defined by

[TABLE]

First suppose that $\lambda_{v}>\nu$. Then, using the the argument in the preceding paragraph, together with the fact that $\lambda_{v}\leq J^{x}_{v}$, we deduce that $\lambda_{v}=J^{x}_{v}$ for all $x\in{\mathds{R}^{d}}$. Thus if $v\in{\Xi_{\mathsf{sm}}}$ is optimal, we must have $\lambda_{v}={\rho_{*}}$. This implies that we can select a ball ${\mathscr{B}}$ such that

[TABLE]

for all sufficiently large $n$. Let ${\breve{\uptau}}=\uptau({\mathscr{B}}^{c})$. By [3, Lemma 2.10 (i)], we have the stochastic representation

[TABLE]

Next we show that that $\Psi_{\mspace{-2.0mu}v}$ vanishes at infinity by using the argument in the proof of Theorem 3.4. The analysis is simpler here. Selecting the same function $\phi$ as in the proof of Theorem 3.4, there exists $R>0$ such that

[TABLE]

Since $\Psi_{\mspace{-2.0mu}v,n}(0)=1$, employing the Harnack inequality we scale $\phi$ so that $\phi>\Psi_{\mspace{-2.0mu}v,n}$ on $B_{R}$ for all $n>R$. The strong maximum principle then shows that $\Psi_{\mspace{-2.0mu}v,n}<\phi$ on ${\mathds{R}^{d}}$.

Thus $\Psi_{\mspace{-2.0mu}v}^{-1}$ is inf-compact, which together with the Lyapunov equation \widetilde{\mathcal{L}}^{\psi_{v}}_{v}\Psi_{\mspace{-2.0mu}v}^{-1}=\bigl{(}c_{v}-{\rho_{*}})\Psi_{\mspace{-2.0mu}v}^{-1} imply that the ground state process is exponentially ergodic. By Lemma 3.2, we then have

[TABLE]

On the other hand, it holds that ${\mathcal{L}}_{v}\Phi_{\mspace{-2.0mu}*}+c_{v}\Phi_{\mspace{-2.0mu}*}\leq{\rho_{*}}\Phi_{\mspace{-2.0mu}*}$, which implies that

[TABLE]

Comparing the functions in Eqs. 51 and 52 using the strong maximum principle, as done in the proof of Lemma 3.2, we deduce that $\Psi_{\mspace{-2.0mu}v}=\Phi_{\mspace{-2.0mu}*}$. Thus $v$ is a measurable selector from the maximizer of Eq. 39.

It remains to address the case $\lambda_{v}\leq\nu$. By [6, Corollary 3.2] there exists a positive constant $\delta$ such that $\lambda_{v}(c_{v}+\delta\mathds{1}_{B_{1}})>\nu$, and $\lambda_{v}(c_{v}+\delta\mathds{1}_{B_{1}})<{\rho_{*}}$. Thus repeating the above argument we obtain

[TABLE]

*Therefore, $v$ cannot be optimal. This completes the proof. *

4 The variational formula on ${\mathds{R}^{d}}$

In this section we establish the variational formula on ${\mathds{R}^{d}}$. As mentioned in Section 1.1, the function ${\mathcal{H}}$ in Eq. 4 plays a very important role in the analysis. To explain how this function arises, let $\operatorname{\mathbb{P}}^{x,t}_{v}$ denote the probability measure on the canonical path space $\{X_{s}\colon 0\leq s\leq t\}$ of the diffusion Eq. 21 under a control $v\in{\Xi_{\mathsf{sm}}}$, and $\widetilde{\operatorname{\mathbb{P}}}^{x,t}_{v}$ the analogous probability measure corresponding to the diffusion

[TABLE]

with ${\varphi_{\mspace{-2.0mu}*}}$ as in Theorem 3.4. By the Cameron–Martin–Girsanov theorem we obtain

[TABLE]

Thus, the relative entropy, or Kullback–Leibner divergence between $\widetilde{\operatorname{\mathbb{P}}}^{x,t}_{v}$ and $\operatorname{\mathbb{P}}^{x,t}_{v}$ takes the form

[TABLE]

Dividing this by $t$, and letting $t\searrow 0$, we see that ${\mathcal{H}}$ is the infinitesimal relative entropy rate.

Recall from Section 1.1 the definition ${\mathcal{Z}}\coloneqq{\mathds{R}^{d}}\times{\mathscr{K}}\times{\mathds{R}^{d}}$, and the use of the single variable $z=(x,\xi,y)\in{\mathcal{Z}}$ in the interest of notational simplicity. Also recall the definitions in Eqs. 5 and 6. Recall the definitions in Eqs. 1 and 2. In analogy to Eq. 16, we define

[TABLE]

The following result plays a central role in this paper.

Proposition 4.1**.**

We have

[TABLE]

*In addition, if ${\mathcal{M}}_{{\mathscr{A}}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}\circ}}({\mathcal{Z}})\subset{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$, then ${{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$ may be replaced by ${\mathcal{P}}({\mathcal{Z}})$ in Eq. 53. *

In the proof of Proposition 4.1 and elsewhere in the paper we use a cut-off function $\chi$ defined as follows (compare this with the function $\breve{\chi}$ in the proof of Theorem 3.4).

Definition 4.2**.**

*Let $\chi\colon\mathds{R}\to\mathds{R}$ be a smooth convex function such that $\chi(s)=s$ for $s\geq 0$, and $\chi(s)=-1$ for $s\leq-2$. Then $\chi^{\prime}$ and $\chi^{\prime\prime}$ are nonnegative and the latter is supported on $(-2,0)$. It is clear that we can choose $\chi$ so that $\chi^{\prime\prime}<1$. We scale this function by defining $\chi_{t}(s)\coloneqq-t+\chi(s+t)$ for $t\in\mathds{R}$. Thus $\chi_{t}(s)=s$ for $s\geq-t$, and $\chi_{t}(s)=-t-1$ for $s\leq-t-2$. Observe that if $-f$ is an inf-compact function then $\chi_{t}(f)+t+1$ is compactly supported by the definition of $\chi$. *

Proof 4.3** (Proof of Proposition 4.1).**

We start with the first equality in Eq. 53. By Eq. 30, we have

[TABLE]

As shown in Theorem 3.4 the twisted process $\tilde{X}$ with extended generator $\widetilde{\mathcal{L}}_{v_{*}}^{{\varphi_{\mspace{-2.0mu}*}}}$ is exponentially ergodic. Let $\eta_{v_{*}}$ denote its invariant probability measure. Since $\frac{\lvert{\varphi_{\mspace{-2.0mu}*}}\rvert}{\Phi_{\mspace{-2.0mu}*}^{-1}}$ vanishes at infinity, and $\Phi_{\mspace{-2.0mu}*}^{-1}$ is a Lyapunov function by Eq. 40, it then follows from Eq. 54, by using the Itô formula and applying [8, Lemma 3.7.2 (ii)], that

[TABLE]

Next, we show that

[TABLE]

We write Eq. 39 as

[TABLE]

and using the identity

[TABLE]

to obtain (compare with Eq. 38)

[TABLE]

Using the function $\chi_{t}$ in Definition 4.2, the identity

[TABLE]

and the definition of ${\mathcal{H}}$, we obtain from Eq. 57 that

[TABLE]

Let $\mu\in{\mathcal{M}}_{{\mathscr{A}}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$, and without loss of generality assume that $\mu\in{{\mathcal{P}}_{\mspace{-3.0mu}\circ}}({\mathcal{Z}})$. The integral of the first term in Eq. 58 with respect to $\mu$ vanishes by the definition of ${\mathcal{M}}_{{\mathscr{A}}}$. Thus, we have

[TABLE]

with $\eta(\cdot)=\int_{{\mathscr{K}}\times{\mathds{R}^{d}}}\mu(\cdot\,,\mathrm{d}{\xi},\mathrm{d}{y})$. Since $\int{\mathcal{H}}\mathrm{d}\eta<\infty$, then taking limits as $t\to\infty$ in Eq. 59, using dominated convergence together with the fact that $\chi^{\prime\prime}_{t}(s)\to 0$ as $t\to\infty$, we see that the right-hand side of Eq. 59 goes to [math]. Also, using Fatou’s lemma and the fact that $\chi^{\prime}_{t}(s)\to 1$ as $t\to\infty$, we obtain from Eq. 59 that

[TABLE]

This proves Eq. 56. Now, if we let

[TABLE]

then

[TABLE]

which implies that $\mu_{*}\in{\mathcal{M}}_{{\mathscr{A}}}$. Then, the second equality in Eq. 55 can be written as

[TABLE]

while the first equality in Eq. 55 together with the fact that $c$ is bounded above and ${\rho_{*}}$ is finite implies that $\mu_{*}\in{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$. Therefore, $\mu_{*}\in{\mathcal{M}}_{{\mathscr{A}}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$, and the first equality in Eq. 53 now follows from Eqs. 56 and 61.

We now turn to the proof of the second equality in Eq. 53. Note that it $\mu\notin{{\mathcal{P}}_{\mspace{-3.0mu}\circ}}({\mathcal{Z}})$ then $F(0,\mu)=-\infty$. On the other hand, if $\mu\notin{\mathcal{M}}_{\mathscr{A}}$ then, as also stated in the proof of Theorem 2.3, $\inf_{g\in{\mathcal{C}}^{2}_{c}({\mathds{R}^{d}})}\,F(g,\mu)=-\infty$. The remaining case is $\mu\in{\mathcal{M}}_{\mathscr{A}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$, for which we have $F(g,\mu)=\int_{{\mathcal{Z}}}{L}(z)\,\mu(\mathrm{d}{z})$, thus proving the equality.

*The second statement of the proposition follows directly from the arguments used above. *

Remark 4.4**.**

One can follow the argument in the proof of [5, Theorem 1.4], using Radon–Nikodym derivatives instead of densities, to show that every maximizing infinitesimal ergodic occupation measure for Eq. 53 has the form

[TABLE]

where $\delta_{y}$ denotes the Dirac mass at $y\in{\mathds{R}^{d}}$, and $\uppi(\mathrm{d}{x},\mathrm{d}{\xi})$ is an optimal ergodic occupation measure of the diffusion associated with operator ${\mathscr{A}}^{*}$ defined by

[TABLE]

*for $(x,\xi)\in{\mathds{R}^{d}}\times{\mathscr{K}}$ and $f\in{\mathcal{C}}^{2}({\mathds{R}^{d}})$. We leave the verification of this assertion to the reader. *

We continue our analysis by investigating conditions on the model parameters which imply that ${\mathcal{M}}_{{\mathscr{A}}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}\circ}}({\mathcal{Z}})\subset{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$. We impose the following hypothesis on the matrix $a$.

{assumption}

The matrix $a$ is bounded and has a uniform modulus of continuity on ${\mathds{R}^{d}}$, and is uniformly non-degenerate in the sense that the minimum eigenvalue of $a$ is bounded away from zero on ${\mathds{R}^{d}}$.

We start with the following lemma, which can be viewed as a generalization of [3, Lemma 3.3]. Section 3, which applies by default throughout the paper, need not be enforced in this lemma.

Lemma 4.5**.**

Consider a linear operator in $\mathds{R}^{d}$, of the form

[TABLE]

and suppose that the matrix $a=\upsigma\upsigma^{\mathsf{T}}$ satisfies Section 4, and the coefficients $b$ and $c$ are locally bounded and measurable. Then, there exists a constant $\widetilde{C}_{0}$ such that any strong positive solution $u\in{\mathscr{W}}_{\text{loc}}^{2,p}(\mathds{R}^{d})$, $p>d$, to the equation

[TABLE]

satisfies

[TABLE]

Proof 4.6**.**

We use scaling. For any fixed $x_{0}\in\mathds{R}^{d}$, with $\lvert x_{0}\rvert\geq 1$, we define

[TABLE]

and the scaled function

[TABLE]

and similarly for the functions $\tilde{a}_{x_{0}}$, $\tilde{b}_{x_{0}}$, and $\tilde{c}_{x_{0}}$. The equation in Eq. 62 then takes the form

[TABLE]

It is clear from the hypotheses that the coefficients of Eq. 63 are bounded in the ball $B_{3}$, with a bound independent of $x_{0}$, and that the modulus of continuity and ellipticity constants of the matrix $\tilde{a}_{x_{0}}$ in $B_{3}$ are independent of $x_{0}$. We follow the argument in [3, Lemma 3.3], which is repeated here for completeness. First, by the Harnack inequality [21, Theorem 9.1], there exists a positive constant $C_{\mathsf{H}}$ independent of the point $x_{0}$ chosen, such that $\tilde{u}_{x_{0}}(y)\leq C_{\mathsf{H}}\,\tilde{u}_{x_{0}}(y^{\prime})$ for all $y,y^{\prime}\in B_{2}$. Let

[TABLE]

By a well known a priori estimate [16, Lemma 5.3], there exists a constant $C_{\mathsf{a}}$, again independent of $x_{0}$, such that,

[TABLE]

where in the last inequality, we used the Harnack property. Clearly then, the resulting constant $\widetilde{C}_{1}$ does not depend on $x_{0}$. Next, invoking Sobolev’s theorem, which asserts the compactness of the embedding {\mathscr{W}}^{2,p}\bigl{(}B_{1}(x_{0})\bigr{)}\hookrightarrow{\mathcal{C}}^{1,r}\bigl{(}B_{1}(x_{0})\bigr{)}, for $p>d$ and $r<1-\frac{d}{p}$ (see [16, Proposition 1.6]), and combining this with Eq. 64, we obtain

[TABLE]

for some constant $\widetilde{C}_{2}$ independent of $x_{0}$. Thus

[TABLE]

Using Eq. 65 and the identity $\nabla{u}(x_{0})=M_{x_{0}}\,\nabla\tilde{u}_{x_{0}}(0)$ for all $x_{0}\in B_{1}^{c}$, we obtain

[TABLE]

*Of course $B_{3}(x_{0})$ is arbitrary. The same is true with any radius, with perhaps a different constant. This completes the proof. *

Remark 4.7**.**

Lemma 4.5* should be compared with similar gradient estimates in the literature. Its benefit is that it matches or exceeds the estimates in [26, Lemma 5.1] and [15, Theorem A.2], without requiring any regularity on the coefficients. *

{assumption}

One of the following holds:

1. (a)

The function $-c$ is inf-compact. 2. (b)

The drift $b$ satisfies

[TABLE] 3. (c)

There exists a constant $\widehat{C}_{0}$ such that (compare this with [4, Theorem 3.1 (b)])

[TABLE]

where ${\varphi_{\mspace{-2.0mu}*}}=\log\Phi_{\mspace{-2.0mu}*}$, and $\Phi_{\mspace{-2.0mu}*}$ is as in Theorem 3.4.

Remark 4.8**.**

Section 4* (c) is not specified in terms of the parameters of the equation. However, Section 4 together with the hypothesis that $\frac{\lvert b\rvert^{2}}{1+\lvert c\rvert}$ is bounded implies Section 4 (c). This is asserted by Lemma 4.5. See also Lemma 4.13 later in this section. *

We have the following estimate concerning the growth of the function $\Phi_{\mspace{-2.0mu}*}$ in Theorem 3.4. This does not require the uniform ellipticity hypothesis in Section 4.

Lemma 4.9**.**

Grant Section 4 part (a) or (b). Then there exists a function $\zeta\colon(0,\infty)\to(0,\infty)$, with $\lim_{r\to\infty}\zeta(r)=\infty$, such that the solution $\Phi_{\mspace{-2.0mu}*}$ in Eq. 39 satisfies

[TABLE]

Proof 4.10**.**

We start with part (a). Let $\alpha\colon(0,\infty)\to(0,\infty)$ be a strictly increasing function, satisfying $\alpha(r)\to\infty$ and $\frac{\alpha(r)}{r}\to 0$ as $r\to\infty$, and

[TABLE]

This is always possible. A specific function satisfying these properties is given by

[TABLE]

Let $c_{1}$ be a constant such that \bigl{\lvert}{\mathcal{L}}_{v_{*}}(\log\lvert x\rvert)\bigr{\rvert}\leq c_{1} for all $\lvert x\rvert>1$. Such a constant exists since $\upsigma$ and $b$ have at most linear growth in $\lvert x\rvert$ by Item (i). We define

[TABLE]

Since the functions $-{\varphi_{\mspace{-2.0mu}*}}$ and $-c$ are inf-compact, it is clear that $\kappa(r)\to\infty$ as $r\to\infty$.

Define the family of functions

[TABLE]

Note that for any $g\in{\mathcal{C}}^{2}({\mathds{R}^{d}})$ we have

[TABLE]

Thus, applying Eq. 70 and the bound \bigl{\lvert}{\mathcal{L}}_{v_{*}}(\log\lvert x\rvert)\bigr{\rvert}\leq c_{1}, we obtain

[TABLE]

Combining Eqs. 54 and 71, and completing the squares, we have

[TABLE]

Recall that $\chi^{\prime}\leq 1$, and $\chi^{\prime\prime}\leq 1$. Choose $r$ large enough so that ${\varphi_{\mspace{-2.0mu}*}}<-1$ on $B_{r}^{c}$. It then follows by the definitions in Eqs. 68 and 69 that {\varphi_{\mspace{-2.0mu}*}}-\chi_{t}\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}h_{r}<0 on $\partial B_{r}$ for all $t\geq 0$. Also, for each $t>0$, the difference {\varphi_{\mspace{-2.0mu}*}}-\chi_{t}\mathbin{\mathchoice{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}{\vbox{\hbox{\scriptscriptstyle\circ}}}}h_{r} is negative outside some compact set by the inf-compactness of $-{\varphi_{\mspace{-2.0mu}*}}$. Note also that $\lvert\nabla h_{r}\rvert\leq\frac{\kappa(r)}{r}$ on $B_{r}^{c}$. Hence Items (i) and 69 imply that there exists $r_{0}$ such the right-hand side of Eq. 72 is negative on $B_{r}^{c}$ for all $r>r_{0}$ and all $t\geq 0$. An application of the strong maximum principle then shows that ${\varphi_{\mspace{-2.0mu}*}}<h_{r}$ on $B_{r}^{c}$ for all $r>r_{0}$.

Now, note that

[TABLE]

Since $\alpha(r)$ is strictly increasing, the inequality Eq. 67 holds with

[TABLE]

This completes the proof under Section 4 (a) .

The proof under part (b) of the assumption is similar. The only difference is that here we use the fact that m_{r}\,\coloneqq\,\sup_{x\in B_{r}^{c}}\,\bigl{(}{\mathcal{L}}_{v_{*}}(\log\lvert x\rvert)\bigr{)}^{-}\to 0 as $t\to\infty$, which is implied by Eq. 66. Thus with $\epsilon>0$ any constant such that ${\rho_{*}}-c>\epsilon$ outside some compact set, we choose $\kappa(r)$ as

[TABLE]

*The rest is completely analogous to the analysis above. This concludes the proof. *

The first part of the theorem which follows is quite technical, but identifies a rather deep property of the ergodic occupation measures of the operator ${\mathscr{A}}$. It shows that under Sections 4 and 4 (a) or (b), or Section 4 (c), if such a measure $\mu$ is feasible for the maximization problem, or in other words, it satisfies $\int_{{\mathcal{Z}}}{L}(z)\,\mu(\mathrm{d}{z})>-\infty$, then it necessarily has “finite average” entropy, that is $\int{\mathcal{H}}\,\mathrm{d}\mu<\infty$, or equivalently, it belongs in the class ${{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$. The proof uses the method of contradiction. We first show that if such a measure $\mu$ is not in the class ${{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$, then the left hand side of Eq. 59 grows at a geometric rate as a function of $t$. Then we obtain a contradiction by evaluating the right-hand side of Eq. 59 using this geometric growth together with the bound in Lemma 4.9.

Theorem 4.11**.**

1. (i)

Under Sections 4 and 4 (a) or (b), or Section 4 (c), we have ${\mathcal{M}}_{{\mathscr{A}}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}\circ}}({\mathcal{Z}})\subset{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$. This of course implies by Proposition 4.1 that

[TABLE] 2. (ii)

Let Section 4 hold, and suppose that

[TABLE]

Then

[TABLE]

Proof 4.12**.**

We first prove part (i) under under Section 4 (a) or (b). We argue by contradiction. Let $\mu\in{\mathcal{M}}_{{\mathscr{A}}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}\circ}}({\mathcal{Z}})$, and suppose that $\mu\notin{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$. As in the proof of Proposition 4.1 we let $\eta(\cdot)=\int_{{\mathscr{K}}\times{\mathds{R}^{d}}}\mu(\cdot\,,\mathrm{d}{\xi},\mathrm{d}{y})$. Let $\mathscr{I}_{1}(t)$ and $\mathscr{I}_{2}(t)$ denote the left and the right-hand side of Eq. 59, respectively, and define

[TABLE]

Then of course $\mathcal{I}(t)\to\infty$ as $t\to\infty$ by the hypothesis. Expanding $\mathscr{I}_{1}(t)$ we see that

[TABLE]

Since $\int{L}\,\mathrm{d}\mu$ is finite, it follows that $\int_{{\mathcal{Z}}}\lvert\upsigma^{\mathsf{T}}y\rvert^{2}\mathrm{d}{\mu}$ and $\int_{{\mathcal{Z}}}\max\{-c,0\}\,\mathrm{d}{\mu}$ are also finite. Moreover, the second assertion and the fact that $c$ is bounded above imply that $\int_{{\mathcal{Z}}}|c|\,\mathrm{d}{\mu}<\infty$. Thus, using the Cauchy–Schwarz inequality in the above display and the fact $|\chi^{\prime}_{t}|$ is bounded, we have

[TABLE]

for some constants $\alpha_{0}(t)$ and $\alpha_{1}(t)$ which are bounded in $t\in[0,\infty)$.

First suppose that over some sequence $t_{n}\to\infty$ we have $\frac{\mathscr{I}_{2}(t_{n})}{\mathscr{I}_{1}(t_{n})}\to\delta<1$ as $n\to\infty$. This implies by Eq. 75 that $\frac{\mathscr{I}_{2}(t_{n})}{\mathcal{I}(t_{n})}\to\delta$. However, if this is the case, then the inequality

[TABLE]

which is implied by Eqs. 59 and 75, contradicts the fact that $\mathcal{I}(t)\to\infty$ as $t\to\infty$. Thus we must have $\liminf_{t\to\infty}\frac{\mathscr{I}_{2}(t)}{\mathscr{I}_{1}(t)}\geq 1$, and same applies to the fraction $\frac{\mathscr{I}_{2}(t)}{\mathcal{I}(t)}$.

Define

[TABLE]

We have $\mathcal{I}(2n)\geq\sum_{k=1}^{n}g_{k}$ for $n\in\mathds{N}$, by definition of these quantities. Recall that $\mathscr{I}_{2}(t)$ is defined as the right-hand side of Eq. 59. Note then that, since $\chi^{\prime\prime}<1$, we have $\mathscr{I}_{2}(2n)<\delta g_{n+1}$ for some $\delta<1$. Therefore, since $\liminf_{t\to\infty}\,\frac{\mathscr{I}_{2}(t)}{\mathcal{I}(t)}\geq 1$, there exists $n_{0}\in\mathds{N}$ such that

[TABLE]

Thus $S_{n+1}-S_{n}=g_{n+1}\geq S_{n}$, which implies that $S_{n+1}\geq 2S_{n}$. This of course means that $S_{n}$ diverges at a geometric rate in $n$, that is, $S_{n}\geq 2^{n-1}S_{1}$. Let $h$ denote the inverse of the map $y\mapsto\zeta(y)\log(1+y)$. Note that ${\mathcal{H}}(x)\leq C(1+\lvert x\rvert^{p})$ for some positive constants $C$ and $p$ by Lemma 4.5 and the hypothesis that $c$ has polynomial growth in Section 3 (ii). Thus, by Lemma 4.9, we obtain

[TABLE]

for all $n\in\mathds{N}$. However, this implies from Eq. 76 that

[TABLE]

for some constant $C^{\prime}$, and we reach a contradiction. Therefore, ${\mathcal{M}}_{{\mathscr{A}}}\cap{{\mathcal{P}}_{\mspace{-3.0mu}\circ}}({\mathcal{Z}})\subset{{\mathcal{P}}_{\mspace{-3.0mu}*}}({\mathcal{Z}})$.

Moving on to the proof under Section 4 (c), we replace the function $\chi_{t}$ in Definition 4.2 by a function $\tilde{\chi}_{t}$ defined as follows. For $t>0$, we let $\tilde{\chi}_{t}$ be a convex ${\mathcal{C}}^{2}(\mathds{R})$ function such that $\tilde{\chi}_{t}(s)=s$ for $s\geq-t$, and $\tilde{\chi}_{t}(s)=\text{constant}$ for $s\leq-t\mathrm{e}^{2}$. Then $\tilde{\chi}^{\prime}_{t}$ and $\tilde{\chi}^{\prime\prime}_{t}$ are nonnegative. In addition, we select $\tilde{\chi}_{t}$ so that $\tilde{\chi}^{\prime\prime}_{t}(s)\leq-\frac{1}{s}$ for $s\in[-t\mathrm{e}^{2},-t]$ and $t\geq 0$. This is always possible. We follow the same analysis as in the proof of Proposition 4.1, with the function $\tilde{\chi}_{t}$ as chosen, and obtain

[TABLE]

where $A_{t}\coloneqq\{x\colon{\varphi_{\mspace{-2.0mu}*}}(x)\leq-t\}$. The integral on the right-hand side of Eq. 77 vanishes as $t\to\infty$ by the hypothesis that $\int c\,\mathrm{d}\mu>-\infty$, so again we obtain Eq. 60 which implies the result. This completes the proof of part (i).

We continue with part (ii). We use a ${\mathcal{C}}^{2}$ convex function $\hat{\chi}_{t}\colon\mathds{R}\to\mathds{R}$, for $t\geq 1$, satisfying $\hat{\chi}_{t}(s)=s$ for $s\leq-t$, $\hat{\chi}^{\prime\prime}_{t}(s)\leq-\frac{1}{s\log\lvert s\rvert}$ for $s<-t$, and $\hat{\chi}_{t}(s)=\text{constant}$ for $s\geq\hat{\zeta}(t)$, for some $\hat{\zeta}(t)<-t$. We let h_{t}(x)=\hat{\chi}_{t}\bigl{(}{\varphi_{\mspace{-2.0mu}*}}(x)\bigr{)}. We may translate ${\varphi_{\mspace{-2.0mu}*}}$ so that it is smaller than $-1$ on ${\mathds{R}^{d}}$. By (58), we have

[TABLE]

We claim that given any $\epsilon>0$ there exists $t>0$ such that $F(h_{t},\mu)\leq{\rho_{*}}+\epsilon$ for all $\mu\in{\mathcal{P}}({\mathcal{Z}})$. This of course suffices to establish Eq. 74.

By Section 3 (iii) there exists $t_{1}>0$ such that the first term on the right-hand side of Eq. 78 is nonpositive for all $t\geq t_{1}$. Also, using the definition of $\hat{\chi}$, we have

[TABLE]

*by the hypothesis, and since $-{\varphi_{\mspace{-2.0mu}*}}$ is inf-compact by Theorem 3.4. This proves the claim, and completes the proof. *

There is a large class of problems which satisfy Eq. 73. It consists of equations with $\lvert b\rvert^{2}+\lvert c\rvert$ having at most linear growth in $\lvert x\rvert$ and $\lvert x\rvert^{-1}\langle b,x\rangle^{-}$ growing no faster than $\lvert c\rvert^{2}$. This fact is stated in the following lemma.

Lemma 4.13**.**

Grant Section 4 and suppose that

[TABLE]

*Then Eq. 73 holds. *

Proof 4.14**.**

We use the function $\chi_{t}$ in Definition 4.2. Let $\tilde{r}>0$ be such that ${\rho_{*}}-c(x,\xi)>\delta>0$ on $B_{\tilde{r}}^{c}\times{\mathscr{K}}$. Note that there exists a constant $C$ such that

[TABLE]

Thus for some $\epsilon>0$ small enough, using Eq. 54, we obtain

[TABLE]

An application of the strong maximum principle then shows that ${\varphi_{\mspace{-2.0mu}*}}(x)\leq\epsilon(\tilde{r}-\lvert x\rvert)^{-}$. Therefore, using Lemma 4.5, we obtain

[TABLE]

*for some constant $C^{\prime}$. *

We next present the variational formula over functions in ${\mathcal{C}}^{2}({\mathds{R}^{d}})$ whose derivatives up to second order have at most polynomial growth in $\lvert x\rvert$. Let ${\mathcal{C}}_{\mathsf{pol}}^{2}({\mathds{R}^{d}})$ denote this space of functions.

Theorem 4.15**.**

Under Section 3 alone, we have

[TABLE]

Under Sections 4 and 4 (a) or (b), we have

[TABLE]

Proof 4.16**.**

By Eqs. 38 and 39 we have

[TABLE]

Since ${\varphi_{\mspace{-2.0mu}*}}\in{\mathcal{C}}^{2}({\mathds{R}^{d}})$, this implies that

[TABLE]

On the other hand, by Theorem 3.4 (d), it follows that for any $g\in{\mathcal{C}}^{2}({\mathds{R}^{d}})$ we have

[TABLE]

which then implies the converse inequality

[TABLE]

This proves Eq. 79.

Concerning Eq. 80, the first equality follows as in the preceding paragraph since ${\varphi_{\mspace{-2.0mu}*}}\in{\mathcal{C}}_{\mathsf{pol}}^{2}({\mathds{R}^{d}})$ by Assumptions 3 (i)–(ii) and 4, and Lemma 4.5. Turning now our attention to the second equality in Eq. 80, recall from the proof of Proposition 4.1 that $\eta_{v_{*}}$ denotes the invariant probability measure of $\widetilde{\mathcal{L}}_{v_{*}}^{{\varphi_{\mspace{-2.0mu}*}}}$. Under Section 4 (a) or (b), Lemma 4.9 shows that $\Phi_{\mspace{-2.0mu}*}^{-1}(x)$ grows faster in $\lvert x\rvert$ than any polynomial. Therefore, $\int_{\mathds{R}^{d}}\lvert x\rvert^{n}\,\eta_{v_{*}}(\mathrm{d}{x})<\infty$ for all $n\in\mathds{N}$ by Eq. 40. Since $\lvert\nabla{\varphi_{\mspace{-2.0mu}*}}(x)\rvert$ has at most polynomial growth, and $b$ has at most linear growth, we obtain

[TABLE]

Continuing, if Eq. 81 holds, then it is standard to show by employing a cut-off function, that

[TABLE]

Let $\mu_{*}\in{\mathcal{M}}_{\mathscr{A}}$ denote the ergodic occupation measure corresponding to $\eta_{v_{*}}$, that is,

[TABLE]

Equation 82* implies that*

[TABLE]

Since

[TABLE]

*the second equality in Eq. 80 then follows by Eqs. 79 and 83. *

5 The risk-sensitive cost minimization problem

Using Lemma 4.5, we can improve the main result in [3] which assumes bounded drift and running cost.

We say that a function $f\colon{\mathcal{X}}\to\mathds{R}$ defined on a locally compact space is coercive, or near-monotone, relative to a constant $\beta\in\mathds{R}$ if there exists a compact set $K$ such that $\inf_{K^{c}}\,f>\beta$. Recall that an admissible control $\xi$ for Eq. 21 is a process $\xi_{t}(\omega)$ which takes values in ${\mathscr{K}}$, is jointly measurable in $(t,\omega)\in[0,\infty)\times\Omega$, and is non-anticipative, that is, for $s<t$, $W_{t}-W_{s}$ is independent of ${\mathfrak{F}}_{s}$ given in Eq. 9. We let ${\Xi}$ denote the class of admissible controls, and $\operatorname{\mathbb{E}}^{x}_{\xi}$ the expectation operator on the canonical space of the process under the control $\xi\in{\Xi}$, conditioned on the process $X$ starting from $x\in\mathds{R}^{d}$ at $t=0$.

Let $c\colon{\mathds{R}^{d}}\times{\mathscr{K}}\to\mathds{R}$ be continuous, and Lipschitz continuous in its first argument uniformly with respect to the second. We define the risk-sensitive penalty by

[TABLE]

and the risk-sensitive optimal values by ${\mathscr{E}}^{x}_{*}\coloneqq\inf_{\xi\in\,{\Xi}}\,{\mathscr{E}}^{x}_{\xi}$, and ${\mathscr{E}}_{*}\coloneqq\inf_{x\in\,{\mathds{R}^{d}}}\,{\mathscr{E}}^{x}_{*}$. Let

[TABLE]

and

[TABLE]

We say that $\widehat{\lambda}_{*}$ is strictly monotone at $c$ on the right if $\widehat{\lambda}_{*}(c+h)>\widehat{\lambda}_{*}(c)$ for all non-trivial nonnegative functions $h$ with compact support.

Proposition 5.1 below improves [3, Proposition 1.1]. We first state the assumptions.

{assumption}

In addition to Section 4 we require the following.

1. (i)

The drift $b$ and running cost $c$ satisfy, for some $\theta\in[0,1)$ and a constant $\kappa_{0}$, the bound

[TABLE]

for all $(x,\xi)\in{\mathds{R}^{d}}\times{\mathscr{K}}$. 2. (ii)

The drift $b$ satisfies

[TABLE]

Proposition 5.1**.**

Grant Section 5, and suppose that $c$ is coercive relative to ${\mathscr{E}}_{*}$. Then the HJB equation

[TABLE]

has a solution $V_{\mspace{-2.0mu}*}\in{\mathcal{C}}^{2}(\mathds{R}^{d})$, satisfying $\inf_{{\mathds{R}^{d}}}\,V_{\mspace{-2.0mu}*}>0$, and the following hold:

1. (a)

${\mathscr{E}}^{x}_{*}={\mathscr{E}}_{*}=\widehat{\lambda}_{*}$* for all $x\in{\mathds{R}^{d}}$.* 2. (b)

Any $v\in{\Xi_{\mathsf{sm}}}$ that satisfies

[TABLE]

a.e. $x\in{\mathds{R}^{d}}$, is stable, and is optimal, that is, ${\mathscr{E}}^{v}_{x}={\mathscr{E}}_{*}$ for all $x\in{\mathds{R}^{d}}$. 3. (c)

It holds that

[TABLE]

for any $v\in{\Xi_{\mathsf{sm}}}$ that satisfies Eq. 86. 4. (d)

If $\widehat{\lambda}_{*}$ is strictly monotone at $c$ on the right, then there exists a unique positive solution to Eq. 85, up to a multiplicative constant, and any optimal $v\in{\Xi_{\mathsf{sm}}}$ satisfies Eq. 86.

Proof 5.2**.**

A modification of [3, Lemma 3.2] (e.g., applying Itô’s formula to the function $f(x)=\lvert x\rvert^{2+2\theta}$) shows that Eq. 84 implies that

[TABLE]

*From this point on, the proof follows as in [3], using Lemma 4.5. Indeed, parts (a) and (b) follow from [3, Theorem 3.4] by using the above estimate and Lemma 4.5. Since $\inf_{{\mathds{R}^{d}}}\,V_{\mspace{-2.0mu}*}>0$, any minimizing selector is recurrent. Moreover, the twisted diffusion corresponding to the minimizing selector is regular. Thus part (c) follows from [3, Theorem 1.5]. In addition, the hypothesis in (d) implies that for any minimizing selector $v$, $\lambda_{v}=\hat{\lambda}_{*}$ is right monotone at $c$ which, in turn, implies the simplicity of the principal eigenvalue by [3, Theorem 1.2]. This also implies the last claim by [3, Lemma 3.6]. *

Acknowledgements

The work of Ari Arapostathis was supported in part by the National Science Foundation through grant DMS-1715210, in part the Army Research Office through grant W911NF-17-1-001, and in part by the Office of Naval Research through grant N00014-16-1-2956 which was approved for public release under DCN #43-5025-19. The research of Anup Biswas was supported in part by an INSPIRE faculty fellowship and DST-SERB grant EMR/2016/004810, while the work of Vivek Borkar was supported by a J. C. Bose Fellowship.