Risk-Sensitive Mean-Field-Type Control

TL;DR

This paper develops a framework for risk-sensitive optimal control of mean-field type stochastic differential equations, deriving infinite-dimensional optimality equations and exploring connections with risk-neutral solutions.

Contribution

It introduces a novel approach to risk-sensitive mean-field control, deriving new optimality equations and analyzing specific cost functionals.

Findings

Derived infinite-dimensional optimality equations for risk-sensitive control

Connected risk-sensitive solutions with risk-neutral cases

Discussed linear-exponentiated quadratic cost functional

Abstract

We study risk-sensitive optimal control of a stochastic differential equation (SDE) of mean-field type, where the coefficients are allowed to depend on some functional of the law as well as the state and control processes. Moreover the risk-sensitive cost functional is also of mean-field type. We derive optimality equations in infinite dimensions connecting dual functions associated with Bellman functional to the adjoint process of the Pontryagin maximum principle. The case of linear-exponentiated quadratic cost and its connection with the risk-neutral solution is discussed.

Equations

$$\underset{\theta \to 0}{\lim }\frac{d}{d\theta }F(m+\theta \tilde{m})={\int }_{\xi \in {\mathbb{R}}^n}\frac{\partial F}{\partial m}(m)(\xi )\tilde{m}(\xi )d\xi ,$$

$$dx(t)=g(x,m_v,v)dt+\sigma (x(t))dw(t),x(0)=x_0,$$

$$A\varphi (x)=-\frac{1}{2}\sum _{i,j}{a}_{ij}(x)\frac{{\partial }^2\varphi (x)}{\partial x_i\partial x_j},$$

$$A^{\ast }\varphi (x)=-\frac{1}{2}\sum _{i,j}\frac{{\partial }^2[a_{ij}(x)\varphi (x)]}{\partial x_i\partial x_j}.$$

$$=\mathbb{E}\left[{\int }_0^{T}f(x,m_v,v)dt+h(x(T),m_v(T))\right].$$

$$J^{\alpha }(v(.))=\mathbb{E}{e}^{\alpha \left[{\int }_0^{T}f(x(t),m_v(t),v(t))dt+h(x(T),m_v(T))\right]}$$

$$\frac{J^{\alpha }(v(.))-1}{\alpha }\to J(v(.)),\text{mbox}as\alpha \to 0.$$

$$\frac{1}{\alpha }{\log }_{10}(J^{\alpha })\to J(v(.)),\text{mbox}as\alpha \to 0.$$

$$H(x,m,q)={\inf }_v\{f(x,m,v)+qg(x,m,v)\}$$

$$\frac{\partial m_{v}}{\partial t}+A^{*}m_{v}+div(g(x,m_{v},v(\cdot))m_{v})=0,\$$

$$J(v(.))={\int }_0^T{\int }_{x\in {\mathbb{R}}^n}f(x,m_v(t),v(t))m_v(x,t)dxdt$$

$$+\int h(x,m_v(T))m_v(x,T)dx.$$

$$u(x,T)=h(x,m)+{\int }_{{\mathbb{R}}^n}\frac{\partial }{\partial m}h(\xi ,m)(x)m(\xi ,T)d\xi$$

$$-\frac{\partial u}{\partial t}+Au=H(x,m,D_{x}u(x))$$

$$+{\int }_{{\mathbb{R}}^n}\frac{\partial }{\partial m}H(\xi ,m,D_{x}u(\xi ))(x)m(\xi ,t)d\xi$$

$$\frac{\partial m}{\partial t}+A^{\ast }m+div(G(x,m,D_{x}u)m)=0,$$

$$m(0,x)=m_0(x).$$

$$u(x,T)=h(x),$$

$$-\frac{\partial u}{\partial t}+Au=H(x,D_{x}u(x)),$$

$$\frac{\partial m}{\partial t}+A^{\ast }m+div(G(x,D_{x}u)m)=0,$$

$$m(0,x)=m_0(x).$$

$$J^{\alpha }(v(\cdot ))=\mathbb{E}{e}^{\alpha \left[{\int }_0^{T}f(x(t),v(t))dt+h(x(T))\right]},$$

$$dx(t)=g(x(t),v(t))+\sigma (x(t))dw(t),x(0)=x_0,$$

$$dz=f(x(t),v(t))dt,z(0)=0,$$

$$J^{\alpha }(v(\cdot ))=\mathbb{E}{e}^{\alpha \left[z(T)+h(x(T))\right]}.$$

$$dx=gds+\sigma (x)dw,x(t)=x$$

$$dz=f(x(s),v(s))ds,z(t)=z.$$

$$-\frac{\partial {\Phi }^{\alpha }}{\partial t}+A{\Phi }^{\alpha }={\inf }_v{D}_x{\Phi }^{\alpha }g(x,v)+\frac{\partial {\Phi }^{\alpha }}{\partial z}(x,v),$$

$${\Phi }^{\alpha }(x,z,T)=e^{\alpha [z+h(x)]}.$$

$$-\frac{\partial u^{\alpha }}{\partial t}+A{u}^{\alpha }={\inf }_v{D}_x{\Phi }^{\alpha }.g(x,v)+\alpha u^{\alpha }f(x,v),$$

$$u^{\alpha }(x,T)=e^{\alpha h(x)}.$$

$$dx=g(x,m_v,v)dt+\sigma (x)dw,x(0)=x_0,$$

$$dz=f(x,m_v,v)dt,z(0)=0.$$

$$m_v(x,t)={\int }_{\mathbb{R}}{\mu }_v(x,z,t)dz.$$

$$\begin{array}[]{lll}\frac{\partial\mu_{v}}{\partial t}+A^{*}\mu_{v}+div(g(x,m_{v},v(x,z,t))\mu_{v})\\ +\frac{\partial}{\partial z}[f(x,m_{v},v(x,z))\mu_{v}]=0,\\ \\ \mu_{v}(0,x,z)=m_{0}(x)\otimes\delta_{0}(z)\end{array}$$

$$J^{\alpha }(v)={\int }_{x\in {\mathbb{R}}^n}{\int }_{z\in \mathbb{R}}\mu (x,z,T)e^{\alpha [z+h(x,{\int }_{\mathbb{R}}{\mu }_v(x,z,T)dz]}dzdx$$

Full text

1 Introduction

We consider the mean-field-type control problem with a risk-sensitive performance functional. For mean-field-type control, the approach is generally to use the maximum principle, see for instance [1, 3, 5, 8, 9]. We refer the reader to [7] for a recent survey on the approach. In [2], it is shown that one can introduce a system of dual Hamilton-Jacobi-Bellman and Fokker-Planck equations, dHJB-FP, similar to that introduced by Lasry & Lions [6] to handle risk-sensitive mean-field-type control problem. However, the solution of the dHJB equation is not the value function, but must be interpreted as an adjoint function for a dual control problem. Here, we extend this approach to the risk-sensitive mean-field-type control problem. We then make the connection with the stochastic maximum principle, and study the linear-exponentiated-quadratic case.

We consider functions $f(x,m,v),g(x,m,v)$ where the arguments are $x\in\mathbb{R}^{n}$, $m$ is a probability measure on $\mathbb{R}^{n}$, but we will remain mostly in the regular case (with respect to Lebesgue measure), in which $m$ represents the probability density, assumed to be in $L^{2}(\mathbb{R}^{n})$ and $v$ is a control in $\mathbb{R}^{d}.$ The function $f$ is scalar, and the function $g$ is a vector in $\mathbb{R}^{n}.$ We also consider $\sigma:x\in\mathbb{R}^{n}\mapsto\sigma(x)\in\mathcal{L}(\mathbb{R}^{n};\mathbb{R}^{n}),$ and $h:\ (x,m)\in\mathbb{R}^{n}\times L^{2}(\mathbb{R}^{n})\mapsto h(x,m)\in\mathbb{R}.$ All these functions are smooth. In the case of the differentiability with respect to the measure $m,$ we use the concept of Gateaux differentiability. If $F:L^{2}(\mathbb{R}^{n})\rightarrow\mathbb{R},$ then

[TABLE]

with the functional $\xi\in\mathbb{R}^{n}\mapsto\frac{\partial F}{\partial m}(m)(\xi)\in L^{2}(\mathbb{R}^{n}).$

Consider a probability space $(\Omega,\mathcal{X},P)$ and a filtration $\mathbb{F}=(\mathcal{F}_{t})_{t\geq 0}$, generated by a Wiener process $w(\cdot)$ in $\mathbb{R}^{n}$. The classical mean-field-type control problem is the following: Given a control process $v(\cdot)$ adapted to the filtration $\mathbb{F}$, the corresponding state equation is the McKean-Vlasov equation of the mean-field type:

[TABLE]

in which $m_{v}(x,t)$ is the probability density of the random variable (state) $x(t).$ The initial value $x_{0}$ is a random variable that is independent of the Wiener process $w(\cdot)$. This density is well-defined if the matrix $a(x)=\sigma(x)\sigma^{*}(x)$ is invertible. We define the second order differential operator

[TABLE]

and its adjoint

[TABLE]

Next define the cumulative expected cost functional $J(v(.))$ as

[TABLE]

The risk-neutral mean-field-type control problem is to minimize $J(v(\cdot)).$

In this paper we consider a risk-sensitive cost functional, which means that we replace (2) by

[TABLE]

in which $\alpha$ is a real number, representing the risk-sensitivity index of the decision-maker. When $\alpha>0,$ it models a risk-averse decision, when $\alpha<0$ a risk-seeker individual.

Note that

[TABLE]

Equivalently,

[TABLE]

So the case (2) is considered as representing the risk-neutral situation corresponding to $\alpha=0$.

From now on, we shall assume that $\alpha>0.$ The case $\alpha<0$ is examined using the same methodology by change $f\rightarrow-f,$ and $h\rightarrow-h.$

2 Risk-Neutral Case

We define the risk-neutral Hamiltonian $H:\,\,(x,m,q)\in\mathbb{R}^{n}\times L^{2}(\mathbb{R}^{n})\times\mathbb{R}^{n}\mapsto H(x,m,q)\in\mathbb{R}$ as

[TABLE]

and the optimal value of $v$ is denoted by $v^{*}(x,m,q).$ We then set $G(x,m,q)=g(x,m,v^{*}(x,m,q)).$

The mean-field-type control problem is easily transformed into a stochastic control problem for a higher dimensional state, which is the probability density $m_{v}(\cdot)$. It is the solution of the Fokker-Planck equation

[TABLE]

and $m_{v}(0,x)=m_{0}(x),$ in which $m_{0}(x)$ is the probability density of the initial value $x_{0}.$ The objective functional $J(v(.))$ can be written as

[TABLE]

The adjoint system of optimality associated with (4) and (2) is given by

[TABLE]

The optimal feedback control is $v^{*}(x,t)=v^{*}(x,m,D_{x}u)$. When the functions $h,f,g$ are mean-field free, i.e., do not depend on $m$ then the equations (6) and (7) reduces to standard Hamilton-Jacobi-Bellman equation in $x$ and a Fokker-Planck-Kolmogorov equation:

[TABLE]

In this case, $u$ can be interpreted as the value function, the optimal feedback $v^{*}(x,t)=v^{*}(x,D_{x}u(x,t))$ is time consistent, which means that it does not depend on the initial condition of the dynamic system (1), whereas when the system is coupled, it does.

3 Risk-Sensitive Case

3.1 Mean-Field Free Case

Let us consider the problem of optimizing

[TABLE]

subject to the state dynamics

[TABLE]

To be able to apply the optimality principle we have to introduce a second state equation, namely

[TABLE]

and we then get

[TABLE]

In this way the functional involves only the final state, but the state is now augmented. The new state is the pair $(x(t),z(t))$. However, we are in the standard situation, in which we can apply Dynamic Programming. Introduce the family of problems

[TABLE]

We denote the solution by x_{x,z,t}(s),z_{x,z,t}(s),\ and set $J^{\alpha}_{x,z,t}(v)=\mathbb{E}e^{\alpha[z_{x,z,t}(T)+h(x_{x,z,t}(T))]}.$ We define \Phi^{\alpha}({x,z,t})=\hbox{\rm inf,}_{v(.)}\ J^{\alpha}_{x,z,t}(v). We can then write the Bellman equation:

[TABLE]

The above system can be solved by separation of variables as follows: $\Phi^{\alpha}(x,z,t)=e^{\alpha z}u^{\alpha}(x,t)$ with $u^{\alpha}(x,t)$ solution of

[TABLE]

We see easily that $\frac{u^{\alpha}-1}{\alpha}\rightarrow u$ as $\alpha\rightarrow 0,$ where the function $u$ is the solution of the risk-neutral system (10)-(11). The optimal control is obtained by a feedback depending on the state $x$, but not on the state $z.$

3.2 Mean-Field Dependence

We now turn to the risk-sensitive mean-field-type control problem with (1) and (3). We introduce again a new state $z(t)$ with the mean-field term:

[TABLE]

We have to consider a feedback $v(x,z)$ depending on the full state $(x,z)$. The simplification which occurred in the case without mean-field, namely the optimal feedback was depending on the state $x$ only, does not extend in the current context. However, we still consider that the probability $m_{v}(t)$ entering in the functions $f$ and $g$ is the probability density of $x(t)$ and not the joint probability distribution $\mu_{v}(x,z,t)$ of the pair $(x(t),z(t))$. Therefore,

[TABLE]

The joint probability distribution $\mu_{v}(x,z,t)$ of the pair $(x(t),z(t))$ solves the degenerate Fokker-Planck-Kolmogorov equation

[TABLE]

and we can write the cost functional as

[TABLE]

We can apply the general theory by adapting the system (6) (7) and (8). We introduce the Hamiltonian of the augmented state as

[TABLE]

reserving the terminology $H(x,m,q)=\tilde{H}(x,m,q,1).$ The optimal feedback control is denoted by $v^{*}(x,m,q,\rho)$ and we set $\tilde{F}(x,m,q,\rho)=f(x,m,v^{*}(x,m,q,\rho)),$ $\tilde{G}(x,m,q,\rho)=g(x,m,v^{*}(x,m,q,\rho)).$

We also set $v^{*}(x,m,q)=v^{*}(x,m,q,1),$

[TABLE]

The risk-sensitive adjoint system is

[TABLE]

3.3 Transformation of the equation

We aim to transform the system (26), (27), (28). We introduce $\chi(x,z,t)$ defined by $\chi(x,z,t)=\frac{\partial u}{\partial z}(x,z,t).$

We differentiate the equation in $u$, in (27), with respect to $z$. Taking account of the fact that the integrals depend only on $x,$ we get the relation

[TABLE]

We have the following result

Lemma 1**.**

The function $(x,z,t)\mapsto\chi(x,z,t)$ is positive: $\chi>0.$

Proof.

We proceed only formally, since the assumptions have not been stated. The equation (32) is a linear parabolic equation in $\chi$, with no right-hand side and strictly positive final condition. This implies the result.

This allows to assert that

[TABLE]

[TABLE]

[TABLE]

So we can write the system

[TABLE]

Remark 1**.**

Note formally that $u^{\alpha},\chi^{\alpha},\mu^{\alpha},m^{\alpha}$ the solution of the system (33),(34), (35), (36), (37), (39) satisfy

[TABLE]

where the pair $(u,m)$ is the solution of the risk-neutral system (6), (7), (8).

4 Stochastic Maximum Principle

We can derive from the system (33)- (39) a stochastic maximum principle. We use the following notation: $X(t),Z(t)$ represent the optimal states and $V(t)$ represent the optimal control. The probability distribution of $X(t)$ is denoted by $\mathbb{P}_{X(t)}.$ We shall define the adjoint processes by $Y(t)=\frac{D_{x}u(X(t),Z(t),t)}{\chi},\ \eta(t)=\chi(X(t),Z(t),t).$

In fact the real adjoint process is $Y.$ Following the standard notation of stochastic maximum principle, the Hamiltonian is written as

[TABLE]

and by definition of $F,G,$

[TABLE]

[TABLE]

In order to state the stochastic maximum principle we compute the Itô differential of $Y(t).$ We apply Ito’s formula to the function $\frac{D_{x}u(X(t),Z(t),t)}{\chi}.$ After tedious calculations, we obtain

[TABLE]

Set $\eta(t)=\chi(X(t),Z(t),t)=:\chi(t).$ Using Itô’s formula,

[TABLE]

in which we have set $l(t)=\sigma^{*}D_{x}\log\chi(X(t),Z(t),t).$ Let also define $\Gamma(t)=D^{2}_{x}u.\sigma(X(t),Z(t),t).$

[TABLE]

The processes $l(t)$ and $\Gamma(t)$ are defined by the fact that $\chi(t)$ and $Y(t)$ are solutions of stochastic backward differential equations.

5 Linear-quadratic risk-sensitive case

5.1 Mean-Field Free Case

Here we assume that $\beta=0$ and f(x,v)=x^{*}Qx+v^{*}Rv,\ g(x,v)=Ax+Bv,\ $h(x)=x^{*}Q_{T}x,\ \sigma(x)=\sigma$

We look for a solution as follows $u^{\alpha}(x,t)=e^{\alpha[x^{*}\Pi(t)x+\rho(t)]},$ where

[TABLE]

Of course, the Riccati equation (40) may fail to have a solution.

5.2 Mean-Field Dependent Case

For linear-quadratic setup we want to solve the following problem (we only consider the one-dimensional case):

[TABLE]

With $g(x,m,v):=ax+bv,\,\,\,f(x,m,v):=\frac{1}{2}v^{2},\,\,\,$ $h(x,m):=\frac{1}{2}x^{2}+\beta\int ym(dy),\,\,\,\sigma(x):=\sigma.$ Note that $g,f,\sigma$ are independent of $m.$ The corresponding Hamiltonian

[TABLE]

where the optimal control is

[TABLE]

noting that, by Lemma 1 or (57) below, $D_{z}u>0.$

The optimal state $(x(t),z(t))$ solves

[TABLE]

Its associated infinitesimal generator is

[TABLE]

5.2.1 The adjoint function $u$

The adjoint function $u$ is the solution of

[TABLE]

with terminal value

[TABLE]

In terms of the process $(x(t),z(t))$ given in (52) we have

[TABLE]

5.2.2 The function $D_{z}u$

Differentiating (54) w.r.t. $z$ we obtain the following PDE for $\chi:=D_{z}u$:

[TABLE]

In terms of the process $(x(t),z(t))$ given in (52) we have

[TABLE]

5.2.3 The function $D_{x}u$

Differentiating (54) w.r.t. $x$ we obtain the following PDE for

$\varphi:=D_{x}u$, (the equality $D_{x}\chi=D_{z}\varphi$ is used in the calculation).

[TABLE]

In terms of the process $(x(t),z(t))$ given in (52) we have

[TABLE]

5.2.4 Characterization of the optimal control

Using (61) and (58), by Itô’s formula, the process $p:=\frac{D_{x}u}{D_{z}u}=\frac{\varphi}{\chi}$ satisfies

[TABLE]

where

[TABLE]

and

[TABLE]

The process $(p,q,l)$ has an explicit solution in terms of $\chi(0)$ and deterministic function $\pi,\omega$ and is given by p(t)=\pi(t)x(t)+\beta\omega(t)\frac{\chi(0)}{\chi(t)},\ $q(t)=\sigma\pi(t)-\beta l(t)\omega(t)\chi(0)\chi^{-1}(t).$ where $\pi,\omega$ solve Riccati equations.

[TABLE]

The expected value $y(t)=E[x(t)]$ of the optimal state solves the ODE

[TABLE]

The optimal value of the problem is $v^{\alpha}:=\frac{D_{z}u(0,x_{0},0)}{\alpha}=\frac{\chi(0)}{\alpha}.$

5.3 Approximation of the risk-sensitive value

In this section we assume that $\beta\neq 0$ is small. In the previous section we provided an explicit solution of the value function in terms of $\frac{\chi(0)}{\alpha}=\mathbb{E}\phi^{\alpha}.$ However, $\chi(0)$ needs to be calculated. When $\beta=0$ this was explicitly computed from $\pi(0),\omega(0)$ (which were denoted $\Pi(0),\rho(0)$). The optimal value of the adjoint function is $u_{0}(x,z,t)=e^{\alpha(z+\frac{1}{2}\Pi(t)x^{2}+\rho(t))}.$

Now, when $\beta\neq 0$ we aim to approximate u_{\beta}(T):=\mathbb{E}\phi^{\alpha}_{T}=e^{\alpha\beta y(T)}\mathbb{E}\Big{[}e^{\alpha[z(T)+\frac{1}{2}x^{2}(T)]}\Big{]}. For $\beta$ small enough we test the ansatz $u_{\beta}(x,z,t)=e^{\alpha\beta y(T)}\tilde{u}(x,z,t)$ where $y(T)=y_{\beta}(T)\sim y_{0}(T)=\bar{x}_{0}e^{\int_{0}^{T}[a-b^{2}\Pi(t)]dt},$

[TABLE]

which is expressed as $\int_{x}u_{0}(x,0,0)m_{0}(x)dx=\int_{x}e^{\frac{1}{2}\Pi(0)x^{2}+\rho(0)}m_{0}(x)dx.$

The function $\tilde{u}$ solves the partial differential equation

[TABLE]

and $\tilde{u}(x,z,T)=e^{\alpha(z+\frac{1}{2}x^{2})}+\alpha\beta\lambda(T),$ where $\lambda(T)=\mathbb{E}_{\mu(.,T)}e^{\alpha(z(T)+\frac{1}{2}x^{2}(T)}.$ We will check that $\tilde{u}(x,z,t)\sim u_{0}(x,z,t)+\alpha\beta x\omega(t)\lambda(T)=\hat{u}(x,z,t),$ for a convenient function of time $\omega(t)$ to be determined. By choosing $\omega(t)$ as

[TABLE]

we easily check that

[TABLE]

This means that the function $\hat{u}$ is consistent with (65) around $t=T.$ When $\beta^{2}$ can be neglected, the approximation of $\tilde{u}$ by $\hat{u}$ becomes exact. If we accept this approximation, we still need to fix the value of $y(T)$ and $\lambda(T).$ This involves calculations using the probability measure $\mu$ which solve the Fokker-Planck-Kolmogorov equation

[TABLE]

From the approximation above it follows that the ratios $\frac{D_{x}u}{D_{z}u}$ and $\frac{D_{x}\hat{u}}{D_{z}\hat{u}}$ are close to each other:

[TABLE]

Neglecting the term in $\beta$ of the last term $\frac{D_{x}u}{D_{z}u}\sim\Pi(t)x$ and hence $\mu$ will be approximated by $\hat{u}$ as

[TABLE]

and $\mu(x,z,0)=\hat{\mu}(x,z,0)=m_{0}(x)\otimes\delta_{0}(z).$ We immediately deduce that the expected value of $x(t)$ as $y(t)=\int_{x\in\mathbb{R}}\int_{z\in\mathbb{R}}x\mu(x,z,t)dxdz$ is solution of the ordinary differential equation

[TABLE]

Next, writing

[TABLE]

and testing with $\hat{\mu}$ we obtain

[TABLE]

This completes the approximation in $\beta.$

6 Conclusion

In this paper we have developed a risk-sensitive mean-field-type optimal control framework. We have considered performance functionals and coefficients that are allowed to depend on some functional of the law as well as the state and control processes. We derived optimality equations in infinite dimensions connecting dual functions associated with Bellman functional to the adjoint processes of the Pontryagin stochastic maximum principle.

Acknowledgement

The first author is supported by grants from the National Science Foundation (1303775 and 1612880), the Research Grants Council of the Hong Kong Special Administrative Region (City U 500113 and 11303316). The research of the second author is supported by grants from the Swedish Research Council. The research of the third author is supported by U.S. Air Force Office of Scientific Research.