Large Sample Mean-Field Stochastic Optimization

TL;DR

This paper analyzes mean-field stochastic optimization problems with finite samples, proving convergence of solutions to a limiting problem characterized by a nonlinear Fokker-Planck-Kolmogorov equation, and relates it to neural network training.

Contribution

It establishes the existence of optimal relaxed controls for finite samples and proves their convergence to the mean-field limit using $ extGamma$-convergence, linking to neural network training.

Findings

Convergence of finite sample solutions to the mean-field limit.

Connection between optimization minimizers and neural network weights.

Existence of optimal relaxed controls in finite sample scenarios.

Abstract

We study a class of sampled stochastic optimization problems, where the underlying state process has diffusive dynamics of the mean-field type. We establish the existence of optimal relaxed controls when the sample set has finite size. The core of our paper is to prove, via $\Gamma$-convergence, that the minimizer of the finite sample relaxed problem converges to that of the limiting optimization problem. We connect the limit of the sampled objective functional to the unique solution, in the trajectory sense, of a nonlinear Fokker-Planck-Kolmogorov (FPK) equation in a random environment. We highlight the connection between the minimizers of our optimization problems and the optimal training weights of a deep residual neural network.

Equations

$$d{X}^{\theta ,i}(t)=f\left(t,\theta (t),X^{\theta ,i}(t),\frac{1}{N}\sum _{j=1}^N\rho (X^{\theta ,j}(t))\right)dt+{\epsilon }^{i}d{W}^i(t),$$

$$\underset{\theta \in {\mathbb{U}}^{\mathbb{P},\mathbb{F}}}{\inf }{J}_N(\theta ):=\underset{\theta \in {\mathbb{U}}^{\mathbb{P},\mathbb{F}}}{\inf }\mathbb{E}\left[L_N({\tilde{X}}^{\theta }(T),\tilde{Y}(0))+{\int }_0^T{R}_N(\theta (t),{\theta }^{\prime }(t);{\tilde{X}}^{\theta }(t),\tilde{Y}(0))dt\right].$$

$$|f(t,{\theta }_1,x,\eta )-f(t,{\theta }_2,y,\xi )|\le [f{]}_{\mathrm{Lip}}[|{\theta }_1-{\theta }_2|+|x-y|+|\eta -\xi |],$$

$${\zeta }^i:=(X^i(0),Y^i(0))\in {\Xi }_K,$$

$$L_N(\tilde{x},\tilde{y}):=\frac{1}{N}\sum _{i=1}^{N}L(x^i,y^i),R_N(\theta ,{\theta }^{\prime };\tilde{x},\tilde{y}):=\frac{1}{N}\sum _{i=1}^{N}R(\theta ,{\theta }^{\prime };x^i,y^i).$$

$$L(x,y):=\alpha |x-y{|}^2,R(\theta ,{\theta }^{\prime };x,y):={\lambda }_1|\theta {|}^2+{\lambda }_2|{\theta }^{\prime }{|}^2+\beta |x-y{|}^2,$$

$${\mathbb{U}}^{\mathbb{P},\mathbb{F}}:=\left\{\theta \in L^2(\Omega ;{\mathcal{H}}_m^1);\theta \text{leavevmode}\mathrm{is}\mathbb{F}\text{mbox}-\mathrm{adapted}\text{leavevmode}\mathrm{and}\text{leavevmode}\theta \in \Theta ,\text{leavevmode}a.s.\text{leavevmode}\mathrm{on}\text{leavevmode}(0,T)\times \Omega \right\}.$$

$$J_N(Q):={\mathbb{E}}^Q\left[L_N({\tilde{X}}^{\theta }(T),\tilde{Y}(0))+{\int }_0^T{R}_N(\theta (t),{\theta }^{\prime }(t);{\tilde{X}}^{\theta }(t),\tilde{Y}(0))dt\right],$$

$${\alpha }_N:=\underset{Q\in \mathcal{Q}(\nu )}{\inf }{J}_N(Q).$$

$${\Omega }_{\infty }:={\Xi }_K^{\mathbb{N}}\times {\mathcal{C}}_p^{\mathbb{N}}\times {\mathcal{H}}_m^1,{\mathcal{F}}_{\infty }:=\mathcal{B}({\Omega }_{\infty }),$$

$$d_{\infty }((\gamma ,w,\vartheta ),(\hat{\gamma },\hat{w},\hat{\vartheta })):=d_1(\gamma ,\hat{\gamma })+d_2(w,\hat{w})+d_3(\vartheta ,\hat{\vartheta }).$$

$$d_1(\gamma ,\hat{\gamma })$$

$$d_2(w,\hat{w})$$

$$d_3(\vartheta ,\hat{\vartheta })$$

$$d{X}_N^i(t)=f\left(t,{\theta }_N(t),X_N^i(t),\frac{1}{N}\sum _{j=1}^N\rho (X_N^j(t))\right)dt+{\epsilon }^{i}d{W}_N^i(t).$$

$${\mu }^N(t):=\frac{1}{N}\sum _{i=1}^N{\delta }_{({\xi }_N^i,X_N^i(t))},\mathrm{for}t\in [0,T].$$

$${\mathbb{Q}}^N:=Q_N\circ ({\mu }^N(0),{\theta }_N,{\mu }^N{)}^{-1}.$$

$${\mathcal{A}}^{\theta ,\eta }\phi (s,e)$$

$$Q_N\circ ({\mu }^N(0),{\theta }_N{)}^{-1}\Rightarrow {\vartheta }_0,N\to \infty .$$

$$⟨\hat{\mu }(t),\phi (t)⟩-⟨\hat{\mu }(0),\phi (0)⟩-{\int }_0^t⟨\hat{\mu }(s),{\mathcal{A}}^{\hat{\theta },⟨\hat{\mu }(s),\rho ⟩}\phi (s)⟩ds=0,\forall \text{leavevmode}\phi \in \mathcal{D}.$$

$${\mathbb{Q}}_{\mu }^N:=Q_N\circ ({\mu }^N{)}^{-1}.$$

$$\underset{M\to \infty }{\lim }\underset{N\ge 1}{\sup }{\mathbb{Q}}_{\mu }^N\left(\left\{\vartheta \in \hat{S};\underset{t\in [0,T]}{\sup }{\int }_E|e{|}^{2+ϵ}\vartheta (t,de)>M\right\}\right)=0,$$

$$\underset{\delta \to 0}{\lim }\underset{N\ge 1}{\sup }{\mathbb{Q}}_{\mu }^N\left(\left\{\vartheta \in \hat{S};\underset{|t-s|\le \delta }{\sup }{\mathcal{W}}_{E,2}(\vartheta (t),\vartheta (s))>\epsilon \right\}\right)=0.$$

$$d_{\hat{S}}(\vartheta ,\hat{\vartheta }):=\underset{t\in [0,T]}{\sup }{\mathcal{W}}_{E,2}(\vartheta (t),\hat{\vartheta }(t)),\vartheta ,\hat{\vartheta }\in \hat{S}.$$

$${\int }_{\tilde{S}}{d}_{\hat{S}}^2(\vartheta ,\hat{\vartheta }){\mathbb{Q}}_{\mu }^N(d\vartheta )={\mathbb{E}}^{Q_N}\left[d_{\hat{S}}^2({\mu }^N,\hat{\vartheta })\right]={\mathbb{E}}^{Q_N}\left[\underset{t\in [0,T]}{\sup }{\mathcal{W}}_{E,2}^2({\mu }^N(t),\hat{\vartheta }(t))\right]$$

$$\le {\mathbb{E}}^{Q_N}\left[\underset{t\in [0,T]}{\sup }{\int }_{E\times E}{\left|e-\hat{e}\right|}^2{\mu }^N(t,de)\hat{\vartheta }(t,d\hat{e})\right]\le 2{\mathbb{E}}^{Q_N}\left[\underset{t\in [0,T]}{\sup }{\int }_E{\left|e\right|}^2{\mu }^N(t,de)\right]+2\underset{t\in [0,T]}{\sup }{\int }_E{\left|\hat{e}\right|}^2\hat{\vartheta }(t,d\hat{e}).$$

$${\mathbb{E}}^{Q_N}\left[\underset{t\in [0,T]}{\sup }{\int }_E{\left|e\right|}^2{\mu }^N(t,de)\right]\le \frac{1}{N}\sum _{i=1}^N{\mathbb{E}}^{Q_N}\left[\underset{t\in [0,T]}{\sup }{\left|({\xi }_N^i,X_N^i(t))\right|}^2\right]$$

$$\le \frac{1}{N}\sum _{i=1}^N{\mathbb{E}}^{Q_N}\left[{\left|{\xi }_N^i\right|}^2\right]+{\mathbb{E}}^{Q_N}\left[\underset{t\in [0,T]}{\sup }{\left|{\tilde{X}}_N(t)\right|}_N^2\right],$$

$$\underset{N\ge 1}{\sup }{\int }_{\hat{S}}{d}_{\hat{S}}^2(\vartheta ,\hat{\vartheta }){\mathbb{Q}}_{\mu }^N(d\vartheta )$$

$$+2\underset{t\in [0,T]}{\sup }{\int }_E{\left|\hat{e}\right|}^2\hat{\vartheta }(t,d\hat{e})$$

$$<+\infty .$$

$$\underset{R\to \infty }{\lim }\underset{N\ge 1}{\sup }{\int }_{\{\vartheta \in \hat{S};d_{\hat{S}}^2(\vartheta ,\hat{\vartheta })\ge R\}}{d}_{\hat{S}}^2(\vartheta ,\hat{\vartheta }){\mathbb{Q}}_{\mu }^N(d\vartheta )=0.$$

$$\underset{R\to \infty }{\lim }\underset{\vartheta \in \mathcal{C}}{\sup }{\int }_{\{e\in E;|e{|}^2\ge R\}}|e{|}^2\vartheta (t,de)=0.$$

Full text

AMS 2000 subject classifications: 93E35, 93E20, 60F05

Keywords and phrases: Mean-field stochastic optimization; relaxed controls; Fokker-Planck-Kolmogorov equation; Gamma-convergence; deep residual neural networks.

1 Introduction

We investigate a class of sampled control problems, where the underlying state process has stochastic dynamics of the mean-field type, but the control process is the same for all data samples. Given an original probability space $(\Omega,\mathcal{F},\mathbb{F},\mathbb{P})$ with filtration $\mathbb{F}=(\mathcal{F}_{t})_{t\in[0,T]}$ satisfying the usual conditions, we consider a $\theta$-controlled state process $\tilde{X}^{\theta}=(X^{\theta,1}(t),\ldots,X^{\theta,N}(t))_{t\in[0,T]}$. For $i=1,\ldots,N$, $X^{\theta,i}=(X^{\theta,i}(t))_{t\in[0,T]}$ is $d$-dimensional and has dynamics specified by

[TABLE]

with initial condition $X^{\theta,i}(0)=X^{i}(0)$. We assume that $f(t,\theta,x,\eta):[0,T]\times\mathbb{R}^{m}\times\mathbb{R}^{d}\times\mathbb{R}^{q}\to\mathbb{R}^{d}$, $\rho:\mathbb{R}^{d}\to\mathbb{R}^{q}$, and $W^{i}=(W^{i}(t))_{t\in[0,T]}$, $i\in\mathbb{N}$, are independent $p$-dimensional Brownian motions, with $\varepsilon^{i}\in\mathbb{R}^{d\times p}$. We consider a population risk minimization criterion:

[TABLE]

Above, $\tilde{Y}(0):=(Y^{1}(0),\ldots,Y^{N}(0))$ with $Y^{i}(0)$ being a $d$-dimensional random variable for $i=1,\ldots,N$, the notation $\theta^{\prime}(t)$ denotes the first-order weak derivative w.r.t $t$ if $\theta\in{\cal H}_{m}^{1}$, and ${\cal H}_{m}^{p}$ denotes the Sobolev space $W^{1,p}((0,T);\mathbb{R}^{m})$ for $p\in\mathbb{N}$. We also use $\mathbb{U}^{\mathbb{P},\mathbb{F}}$ to denote the admissible strict control set under $\mathbb{P}$, which will be specified in Section 2.1. Further, we assume both the terminal loss function $L_{N}$ and the regularizer function $R_{N}$ to be of quadratic type.

The objective of our paper is to establish that the sequence of minimizers of the sampled objective functionals $(J_{N})_{N=1}^{\infty}$ converges, as $N$ increases, to the minimizer of a limiting objective functional $J$. Because the minimizing sequence of strict controls is not guaranteed to be precompact in $L^{2}(\Omega;{\cal H}_{m}^{1})$, one cannot guarantee the convergence of the sequence of strict control minimizers to the strict control minimizer of the limiting optimization problem. We overcome this challenge by considering relaxed, rather than strict, controls and characterize a global solution to the relaxed control problem. Relaxed controls were first studied by El Karoui et al. (1987). Using Krylov’s Markovian selection theorem, Haussmann and Lepeltier (1990) established the existence of Markovian feedback policies for relaxed optimal control problems. The relaxed stochastic maximum principle was developed, respectively in singular and partially observed optimal control problems, by Bahlali et al. (2007) and Ahmed and Charalambous (2013). More recently, relaxed controls have been applied to analyze the existence of Markovian equilibria and relaxed $\epsilon$-Nash equilibrium of mean field games, both in the presence of idiosyncratic and common noise; see Lacker (2015), Lacker (2016), Carmona et al. (2016) and Carmona and Delarue (2018).

Our main result is to show that, as $N$ gets larger, the sequence of relaxed controls which minimize the finite-dimensional sampled objective functional converges to the relaxed control minimizer of the limiting objective functional. To the best of our knowledge, this convergence result is novel in the literature, and has not been established even in the case where the underlying state dynamics are purely diffusive and have no mean-field interaction.

We next elucidate on the main technical steps and contributions. First, we prove the existence of optimal relaxed solutions to the finite sampled optimization problem stated in Eq. (9) by establishing the precompactness of the corresponding minimizing sequence (c.f. Section 3). For any relaxed control $Q_{N}$ of the finite sample problem with $N$ data, we use the coordinate process consisting of Brownian drivers, input-output samples and control process $\theta_{N}$ to generate a sequence of empirical measure-valued processes $(\mu^{N})_{N=1}^{\infty}$ (see Eq. (14)). We then show that this process admits a limit, which can be characterized by the joint distribution of the initial sample data, the control process and the unique solution of a nonlinear FPK equation in a random environment (c.f. Section 4). We introduce the limiting objective functional $J$, and define it in terms of the joint law of initial data, control process and state process. The main result presented in Theorem 5.1 is as follows: as $N\to\infty$, the sequence of relaxed controls which minimize the sampled objective functional $J_{N}$ converges to the relaxed control which minimizes the limiting objective functional $J$. To prove this result, we show that $J_{N}$ $\Gamma$-converges to $J$ on the space of relaxed controls (c.f. Section 5).

Connections to Deep Residual Neural Network (ResNet). The class of optimization problems considered in our study can be used to characterize the optimal training weights in a deep ResNet fed with many training data. ResNets were first introduced in the influential study of He et al. (2016) to remedy against the degradation problem faced by deep convolutional neural networks, where the training accuracy degrades rapidly as the network depth increases.

Leveraging the optimal control problem formulation of a deep ResNet, first proposed in E (2017), and rigorously analyzed in E et al. (2018), we demonstrate how the stochastic state dynamics with mean-field interactions captures two types of regularization during the training process. The first type is regularization by noise, and consists in injecting noise into the transition function at each layer of the ResNet. This limits the amount of information carried by the units, and acts as a form of regularization because it averages over local neighborhoods of the transition function. The second type of regularization captured by our framework is the so-called batch normalization (Ioffe and Szegedy (2015)). Such technique was designed to remedy against the fact that the distribution of each layer’s inputs changes during training, as the parameters of previous layers change. This is achieved via a normalization step that fixes the means and variances of layer inputs. Such normalization requires sampling inputs from a distribution, and leads to mean-field interactions in the dynamics of the deep ResNet.

There has been a growing literature on optimal control approaches to deep ResNets, modeled as continuous time dynamical systems. The work by Thorpe and van Gennip (2020) provide a discrete-to-continuum Gamma-convergence result for the objective function of such a stylized ResNet, albeit in a deterministic setup. They consider a finite training set and show that the objective function of the ResNet converges, as the number of layers goes to infinity, to a variational problem specified by a set of nonlinear differential equations. Unlike Thorpe and van Gennip (2020) who compare discretized multi-layer and continuous layer ResNets, the starting point of our analysis is a continuous layer ResNet. Our objective is to analyze the relation between a ResNet fed with finitely many training samples and one fed with infinitely many samples. Jabir et al. (2019) use a neural ODE to describe the learning process of a network. They derive the Pontryagin’s optimality principle for the relaxed control problem which yields the weights of a network with finite sample size, and then provide convergence guarantees for the gradient descent algorithm. Cuchiero et al. (2020) also view deep neural networks as discretizations of controlled ODEs.

Some studies have analyzed the performance of neural networks consisting of a single layer. Sirignano and Spiliopoulos (2020) model single-layer networks as limits of the sequence of empirical measures of interacting particles. They derive an evolution equation for the empirical measure, and update parameters through the stochastic gradient descent algorithm (SGD). Chizat and Bach (2018) consider a neural network with a single large hidden layer. They study the performance of a non-convex particle gradient descent algorithm as the number of particles grows large, and assume a quadratic loss function to be minimized.

Other studies have considered deeper neural networks. Mei et al. (2018) study two-layer neural networks, and show that the dynamics of stochastic gradient descent algorithms used for parameter learning can be approximated by a nonlinear partial differential equation (PDE). Rotskoff and Vanden-Eijnden (2019) propose an interacting particle systems model which nests multi-layer neural networks. Therein, to overcome the difficulty of minimizing the training error over the set of parameters, they study the minimum of the loss function over its empirical distribution. Du et al. (2019) study a deep overparameterized ResNet, and prove that gradient descent achieves zero training loss. Sirignano and Spiliopoulos (2022) characterize multi-layer neural networks as the number of hidden layers grows large, and the number of stochastic gradient descent (SGD) iterations grows to infinity. We refer to Mallat (2016) for a mathematical treatment of deep neural networks, with a focus on convolutional architectures.

The remainder of this paper is organized as follows. In Section 2, we formulate the sample-based stochastic control problem and its corresponding relaxed form. In Section 3, we establish the well-posedness of the sampled relaxed control problem for finite sample size. In Section 4, we study the convergence of the sampled objective functional under a suitable Wasserstein metric, as the number of samples grows to infinity. In Section 5, we prove that the sequence of minimizers of sampled objective functionals converges to the minimizer of the limiting objective functional using $\Gamma$-convergence. Section 6 shows how the proposed control framework can be used to model the training process of a deep ResNet. Section 7 concludes. Some proofs and additional auxiliary results are delegated to an Appendix.

2 Sampled Stochastic Control Problem and its Relaxation

Consider the stochastic dynamical system given in Eq. (1). The $\mathbb{F}$-adapted process $\theta=(\theta(t))_{t\in[0,T]}$ is the control strategy, which will be optimally chosen to minimize the objective criterion introduced later in this section.

We impose the following assumptions to ensure that the sampled controlled system described by (1) is well posed:

• (Aε,f,ρ)

1. (i)

there exists a global constant $K>0$ such that $|\varepsilon^{i}|\leq K$ for all $i\in\mathbb{N}$; 2. (ii)

the function $[0,T]\ni t\to f(t,0,0,0)$ is continuous; 3. (iii)

the function $f(t,\theta,x,\eta):[0,T]\times\mathbb{R}^{m}\times\mathbb{R}^{d}\times\mathbb{R}^{q}\to\mathbb{R}^{d}$ is Lipschitz continuous in $(\theta,x,\eta)$ uniformly in $t$, i.e., for $(\theta_{1},x,\eta)$, $(\theta_{2},y,\xi)\in\mathbb{R}^{m}\times\mathbb{R}^{d}\times\mathbb{R}^{q}$,

[TABLE]

where $[f]_{{\rm Lip}}$ is the Lipschitz coefficient of $f$ which is independent of the time variable $t$; 4. (iv)

the function $\rho:\mathbb{R}^{d}\to\mathbb{R}^{q}$ is Lipschitz continuous.

The Lipschitz conditions imposed in (iii)-(iv) of Assumption (Aε,f,ρ) guarantee the existence of a unique strong solution of the stochastic system (1).

2.1 Sampled Control Problem

We begin by formulating the strict sampled optimization problem, and then introduce the corresponding relaxed version. The control process $\theta=(\theta(t))_{t\in[0,T]}$ takes values on $\Theta\subset\mathbb{R}^{m}$ in an admissible set $\mathbb{U}^{\mathbb{P},\mathbb{F}}$, and is obtained by minimizing the population risk criterion given by (2). We assume that

[TABLE]

where $\Xi_{K}:=[-K,K]^{2d}$ for a globally positive constant $K$. This means that the initial samples are assumed to have compact support. For $(\tilde{x},\tilde{y},\theta,\theta^{\prime})\in(\mathbb{R}^{d})^{N}\times(\mathbb{R}^{d})^{N}\times\mathbb{R}^{m}\times\mathbb{R}^{m}$, define

[TABLE]

Above, $L:\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}_{+}$ denotes the terminal loss function, and $R:\mathbb{R}^{m}\times\mathbb{R}^{m}\times\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}_{+}$ is the regularizer of the control problem. We assume $(L,R)$ to be of the quadratic form, i.e., for $\alpha,\lambda_{1},\lambda_{2}>0$ and $\beta\geq 0$,

[TABLE]

for $(x,y,\theta,\theta^{\prime})\in\mathbb{R}^{d}\times\mathbb{R}^{d}\times\mathbb{R}^{m}\times\mathbb{R}^{m}$. Taking the ${\cal H}_{m}^{1}$-regularizer into account, the admissible set $\mathbb{U}^{\mathbb{P},\mathbb{F}}$ is defined as:

[TABLE]

The admissible set $\mathbb{U}^{\mathbb{P},\mathbb{F}}$ is usually referred to as the strict control set under the original probability space. Throughout the paper, we assume that the state space of the parameter process $\theta$ satisfies the following property:

• (AΘ)

$\Theta\subset\mathbb{R}^{m}$ is a compact set, which is not necessarily convex.

The objective of our paper is to establish the convergence of minimizers of the sampled objective functional $J_{N}$ to the minimizer of a limiting objective functional $J$ as $N\to\infty$. We are confronted with two main technical challenges. First, under (AΘ), the problem (2) is a control problem with a non-convex policy space, hence in general a global optimum $\theta^{*}\in\mathbb{U}^{\mathbb{P},\mathbb{F}}$ is not guaranteed to exist. Specifically, the existence of an optimal solution to the optimization problem (2) does not follow from standard compactness techniques used in deterministic optimization problems. Second, even if the set $\Theta$ were convex, the convergence of the sequence of strict control minimizers to the strict control minimizer of the limiting optimization problem is not guaranteed. This is because the associated minimizing sequence of strict controls is, in general, not precompact in $L^{2}(\Omega;{\cal H}_{m}^{1})$.

2.2 Sampled Relaxed Control Problem

We use relaxed controls to bypass the key technical challenges described at the end of the previous section. The set of relaxed controls is the space of probability measures on the data $\zeta=(\zeta^{i})_{i=1}^{\infty}$, the control process $\theta$, and the infinite-dimensional Browian motion $(W^{i})_{i=1}^{\infty}$. This is a convex set, and compact with weak topology if $\Theta$ is compact. As a result, a global solution to the relaxed control problem is always guaranteed to exist. Moreover, the minimizing sequence of relaxed controls is precompact in the space of probability measures under the weak topology. We are then able to establish the convergence of the sequence of (relaxed) minimizers of the sampled objective functional $J_{N}$ to the minimizer of the limiting objective functional $J$, as $N$ grows to infinity. We provide an expression for the limiting objective functional $J$ (see Eq. (60)), and relate its representation to the solution of a FPK equation in a random environment.

We begin by establishing a canonical measurable space $(\Omega_{\infty},\mathcal{F}_{\infty})$ via an infinite product space such that the coordinate process $(\zeta,W,\theta)=((\zeta^{i})_{i=1}^{\infty},(W^{i})_{i=1}^{\infty},\theta)$ can be constructed in the probability space $(\Omega_{\infty},\mathcal{F}_{\infty},Q)$, where $Q$ is referred to as a relaxed control. We consider the natural filtration $\mathbb{F}=(\mathcal{F}^{\zeta,W,\theta}_{t})_{t\in[0,T]}$ generated by $(\zeta,W,\theta)$, which is the completion of the filtration flow $\sigma(\zeta)\vee\sigma(W(s),\theta(s);\ s\leq t)$ for $t\in[0,T]$. Then, the set ${\cal Q}(\nu)$ of relaxed controls is a collection of probability measures $Q$ on $(\Omega_{\infty},\mathcal{F}_{\infty})$ such that, under $(Q,\mathbb{F})$, the initial sample data $\zeta$ has a given law $\nu$; $W$ is a sequence of independent Wiener processes; and $\theta\in\mathbb{U}^{Q,\mathbb{F}}$. We will formally specify ${\cal Q}(\nu)$ in Definition 3.1. Changing the control variable $\theta\in\mathbb{U}^{\mathbb{P},\mathbb{F}}$ in (2) to $Q\in{\cal Q}(\nu)$, the sampled objective functional is given by

[TABLE]

where, for $i=1,\ldots,N$, the state process $(X^{\theta,i}(t))_{t\in[0,T]}$ is the strong solution to Eq. (1) driven by $(\zeta,W,\theta)$. Thus, the relaxed control problem in the finite sample case can be formulated as:

[TABLE]

We call $Q^{*}\in{\cal Q}(\nu)$ an optimal (relaxed) solution of the optimization problem (9) if $J_{N}(Q^{*})=\alpha_{N}$.

Remark 2.1**.**

For a fixed sample size $N$, we only use the first $N$ components of the vector $(\zeta,W)=(\zeta^{i},W^{i})_{i=1}^{\infty}$ in the sampled objective functional $J_{N}(Q)$, where $Q\in{\cal Q}(\nu)$. As $N$ tends to infinity, the sampled objective functional $J_{N}(Q)$ will eventually include all components of $(\zeta,W)=(\zeta^{i},W^{i})_{i=1}^{\infty}$.

3 Optimal Relaxed Solutions of Sampled Optimization Problem

This section establishes the existence of optimal relaxed solutions for the sampled optimization problem (9). We work with the canonical probability space, which needs to be established in terms of an infinite product space. To wit, define

[TABLE]

where ${\cal C}_{d}:=C([0,T];\mathbb{R}^{d})$ denotes the space of continuous functions from $[0,T]$ to $\mathbb{R}^{d}$ equipped with the uniform norm $\|h\|_{T}:=\sup_{t\in[0,T]}|h(t)|$ for any $h\in{\cal C}_{d}$. For $\zeta=(\zeta^{i})_{i=1}^{\infty}$ and $W=(W^{i})_{i=1}^{\infty}$, we use $(\zeta,W,\theta)$ to denote the identity map on $\Omega_{\infty}$. Let $\mathbb{F}=(\mathcal{F}^{\zeta,W,\theta}_{t})_{t\in[0,T]}$ be the complete natural filtration generated by $(\zeta,W,\theta)$. It follows from the Sobolev embedding theorem (see, e.g. Evans (2010)) that $h\in{\cal H}_{m}^{1}$ if and only if $h$ equals ($dt$-a.e.) to an absolutely continuous function whose ordinary derivative (which exists $dt$-a.e.) belongs to ${\cal L}_{m}^{2}$. Here, for $p\geq 1$, ${\cal L}_{m}^{p}:=L^{p}((0,T);\mathbb{R}^{m})$ equipped with the $L^{p}$-norm $\|h\|_{{\cal L}_{m}^{p}}:=\{\int_{0}^{T}|h(t)|^{p}dt\}^{1/p}$ for any $h\in{\cal L}_{m}^{p}$. Then, we treat ${\cal H}_{m}^{1}$ as a subset of ${\cal C}_{m}$. We next endow the space $\Omega_{\infty}$ with the following metric: for $(\gamma,w,\vartheta)$ and $(\hat{\gamma},\hat{w},\hat{\vartheta})\in\Omega_{\infty}$, define

[TABLE]

The above metrics $d_{i}$, for $i=1,2,3$, are given by

[TABLE]

The space $(\Omega_{\infty},d_{\infty})$ is a Polish space (see, e.g. Section 3.8 in Aliprantis and Border (2006), page 89). We next define the relaxed controls formally used in this paper. Let $\nu\in{\cal P}(\Xi_{K}^{\mathbb{N}})$ be a given initial sample law.

Definition 3.1**.**

The set of relaxed controls ${\cal Q}(\nu)$ is defined to be the set of probability measures $Q$ on $(\Omega_{\infty},\mathcal{F}_{\infty})$ satisfying the following three properties:

• (i)

$Q\circ\zeta^{-1}=\nu$;

• (ii)

$W$* consists of a sequence of independent Wiener processes on $(\Omega_{\infty},\mathbb{F},Q)$;*

• (iii)

$\theta$* is an $\mathbb{F}$-adapted and $\Theta$-valued process with $\langle Q,\|\theta\|_{{\cal H}_{m}^{1}}^{2}\rangle:=\mathbb{E}^{Q}[\|\theta\|_{{\cal H}_{m}^{1}}^{2}]<\infty$, i.e., $\theta\in\mathbb{U}^{Q,\mathbb{F}}$.*

For $Q\in{\cal Q}(\nu)$, we refer to $(\zeta,W,\theta)$ as the coordinate process corresponding to $Q$.

The next proposition, whose proof is reported in the Appendix, establishes the existence of optimal relaxed solutions to the stochastic optimization problem (9).

Proposition 3.1**.**

Let (Aε,f,ρ)* and (AΘ)** hold. Then, there exists an optimal solution to the relaxed control problem (9) in the finite sample case.*

4 Limit of Large Sampled Optimization Problem

In this section, we establish the convergence of the sampled objective functional $J_{N}$ given by (8) as the sample size $N$ tends to infinity. We first establish a general convergence result for a class of empirical processes which arise in our sampled controlled dynamic system. The convergence result is then used to (i) establish the limiting behavior of $J_{N}$ in Section 4.3; and (ii) prove the Gamma-convergence of $J_{N}$ to $J$ in Section 5.

4.1 Convergence of Empirical Processes for Large Samples

For $N\in\mathbb{N}$, let $Q_{N}\in{\cal Q}(\nu)$ where $\nu\in{\cal P}(\Xi_{K}^{\mathbb{N}})$ is the initial sample law. Let $(\zeta_{N},W_{N},\theta_{N})$ be the coordinate process corresponding to $Q_{N}$ as in Definition 3.1. Moreover, let $\tilde{X}_{N}=(X_{N}^{1}(t),\ldots,X_{N}^{N}(t))_{t\in[0,T]}$ be a solution of the following SDE:

[TABLE]

In other words, $X_{N}^{i}$ satisfies the SDE (1) driven by $(\zeta_{N},W_{N},\theta_{N})$. Define $E:=\mathbb{R}^{d\times p}\times\mathbb{R}^{d}\times\mathbb{R}^{d}$ and introduce the following empirical measure-valued process given by

[TABLE]

Above, for $i\geq 1$, $\xi_{N}^{i}:=(\varepsilon^{i},Y_{N}^{i}(0))\in\mathbb{R}^{d\times p}\times\mathbb{R}^{d}$. We will show later that, for $N\geq 1$, we can view $\mu^{N}=(\mu^{N}(t))_{t\in[0,T]}$ as a sequence of $\hat{S}:=C([0,T];{\cal P}_{2}(E))$-valued random variables, where ${\cal P}_{p}(E)$ is the $p$-order Wasserstein space with underlying metric space $(E,d_{E})$. We also recall that $C([0,T];{\cal P}_{2}(E))$ is the space of continuous ${\cal P}_{2}(E)$-valued functions defined on $[0,T]$. For $N\geq 1$, we define the following joint distribution:

[TABLE]

The main result of this section is to characterize the limiting behavior of the sequence of joint laws $(\mathbb{Q}^{N})_{N=1}^{\infty}$ (see Theorem 4.1). This is recovered as the unique solution of a FPK equation in a random environment. To start with, let ${\cal D}:=C_{0}^{\infty}(\mathbb{R}^{m})$ be the space of test functions with its dual space given by ${\cal D}^{\prime}$. We introduce a related parameterized operator defined on ${\cal D}$. Formally, for $(\theta,\eta)\in{\cal C}_{m}\times\mathbb{R}^{q}$, and $(s,e)=(s,(\varepsilon,y,x))\in[0,T]\times E$, define

[TABLE]

where $\nabla_{x}\varphi:=(\frac{\partial\varphi}{\partial x_{1}},\ldots,\frac{\partial\varphi}{\partial x_{d}})^{\top}$ and $\nabla_{xx}^{2}\varphi:=(\frac{\partial^{2}\varphi}{\partial x_{k}\partial x_{r}})_{k=1,\ldots,q}^{r=1,\ldots,d}$.

Theorem 4.1**.**

Let (Aε,f,ρ)* and (AΘ)** hold. Suppose further that, for some $\vartheta_{0}\in{\cal P}({\cal P}_{2}(E)\times{\cal C}_{m})$,*

[TABLE]

Then $(\mathbb{Q}^{N})_{N=1}^{\infty}$ defined by (15) converges in ${\cal P}_{2}({\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S})$. Moreover, if the law of a ${\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S}$-valued r.v. $(\hat{\mu}_{0},\hat{\theta},\hat{\mu})$ defined on some probability space $(\hat{\Omega},\hat{\mathcal{F}},\hat{\mathbb{P}})$ is a limit point of $(\mathbb{Q}^{N})_{N=1}^{\infty}$, then, $\hat{\mathbb{P}}$-a.s., $\hat{\mu}$ is the unique solution to the following FPK equation in a random environment: $\hat{\mu}(0)=\hat{\mu}_{0}$, and for $t\in(0,T]$,

[TABLE]

The roadmap of the proof of Theorem 4.1 consists of three steps:

• (i)

We prove the precompactness of the marginal distributions $({\mathbb{Q}}_{\mu}^{N})_{N=1}^{\infty}$ in ${\cal P}_{2}(\hat{S})$, where

[TABLE]

It thus follows from (17) that the sequence of measures $(\mathbb{Q}^{N})_{N=1}^{\infty}$ is tight.

• (ii)

We then prove that, for any weak limit point of a convergent subsequence of $(\mathbb{Q}^{N})_{N=1}^{\infty}$ of the form $\hat{\mathbb{P}}\circ(\hat{\mu}_{0},\hat{\theta},\hat{\mu})^{-1}$, $\hat{\mu}$ is the unique solution of a FPK equation in a random environment with initial condition $\hat{\mu}_{0}$, $\hat{\mathbb{P}}$-a.s..

• (iii)

Finally, we show that $(\mathbb{Q}^{N})_{N=1}^{\infty}$ admits a unique weak limit point.

We next give a lemma (whose proof is reported in the Appendix), which will be used to verify the relative compactness of $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}$ in ${\cal P}(\hat{S})$ needed to prove step (i).

Lemma 4.2**.**

Let (Aε,f,ρ)* and (AΘ)** hold. Let $\epsilon>0$. Then, it holds that*

[TABLE]

and for any $\varepsilon>0$,

[TABLE]

Above, ${\cal W}_{E,2}$ is the quadratic Wasserstein metric on ${\cal P}_{2}(E)$.

The following lemma completes step (i).

Lemma 4.3**.**

Let (Aε,f,ρ)* and (AΘ)** hold. Then, the sequence of marginal distributions $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}$ defined by (19) is relatively compact in ${\cal P}_{2}(\hat{S})$.*

Proof.

We first verify that $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}\subset{\cal P}_{2}(\hat{S})$. Let $e=(\varepsilon,y,x)$ and $\hat{e}=(\hat{\varepsilon},\hat{y},\hat{x})\in E$. We take a measure-valued process $\hat{\vartheta}\in\hat{S}$ satisfying $\sup_{t\in[0,T]}\int_{E}|\hat{e}|^{2}\hat{\vartheta}(t,d\hat{e})<+\infty$. We endow $\hat{S}$ with the metric

[TABLE]

Then, for any $N\geq 1$, we have that

[TABLE]

Since $\hat{\mu}\in\hat{S}$, the 2nd term on the r.h.s. of the inequality (23) is finite. For the 1st term on the r.h.s. of the inequality (23), using (19), it follows that

[TABLE]

where we recall that $\tilde{X}_{N}(t):=(X_{N}^{1}(t),\ldots,X_{N}^{N}(t))$ for $t\in[0,T]$. Using (Aε,f,ρ), Lemma A.1 and noting that $\zeta^{i}\in\Xi_{K}$ for all $i\geq 1$, it follows from (23) that

[TABLE]

This shows that $\mathbb{Q}_{\mu}^{N}\in{\cal P}_{2}({\hat{S}})$ for all $N\geq 1$.

We next prove that $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}\subset{\cal P}_{2}(\hat{S})$ is relatively compact. By Theorem 7.12 in Villani (2003), $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}$ is relatively compact in ${\cal P}_{2}(\hat{S})$ if and only if (I) $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}$ is relative compact in ${\cal P}(\hat{S})$; and (II) $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}$ satisfies the uniform integrability condition, i.e., for some $\hat{\vartheta}\in\hat{S}$,

[TABLE]

• •

The proof of (I): $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}$ is relatively compact in ${\cal P}(\hat{S})$.

By Ascoli’s theorem, a subset ${\cal C}\subset\hat{S}=C([0,T];{\cal P}_{2}(E))$ is relatively compact if (I1): for each $t\in[0,T]$, $\{\vartheta(t);\ \vartheta\in{\cal C}\}\subset{\cal P}_{2}(E)$ is relatively compact; and (I2): ${\cal C}$ is equicontinuous under ${\cal W}_{2}$. Moreover, using again Theorem 7.12 in Villani (2003), (I1) holds if and only if (I11) holds: for each $t\in[0,T]$, $\{\vartheta(t);\ \vartheta\in{\cal C}\}$ is relatively compact in ${\cal P}(E)$; and (I12) holds: the following uniform integrability condition is satisfied:

[TABLE]

Let $\epsilon>0$. For $M,\delta,\varepsilon>0$, define the following subsets of $\hat{S}$:

[TABLE]

Then, for any $\vartheta\in{\cal C}_{1}(M)$ and $t\in[0,T]$, it follows that

[TABLE]

where $B_{R}(0):=\{e\in E;\ |e|\leq R\}$ for $R>0$. On the other hand, it is easy to see that

[TABLE]

By (I11) and (I12), this implies that ${\cal C}_{1}(M)\subset\hat{S}$ satisfies (I1). To continue, fix $\varepsilon>0$, by (20) of Lemma 4.2 there exists $N_{0}=N_{0}(\varepsilon)\geq 1$ and $M_{0}=M_{0}(\varepsilon)\geq 1$ such that $\sup_{N\geq N_{0}}\mathbb{Q}_{\mu}^{N}({\cal C}_{1}^{c}(M_{0}))\leq\frac{\varepsilon}{2}$. Note that $\lim_{M\to\infty}\mathbb{Q}_{\mu}^{N}({\cal C}_{1}^{c}(M))=0$ for any $N\geq 1$ by (20). It follows that there exists $M_{1}=M_{1}(\varepsilon)$ large enough such that $\sup_{1\leq N\leq N_{0}}\mathbb{Q}_{\mu}^{N}({\cal C}_{1}^{c}(M_{1}))\leq\frac{\varepsilon}{2N_{0}}$. Then, let $M:=M_{0}\vee M_{1}$, and hence $\sup_{N\geq 1}\mathbb{Q}_{\mu}^{N}({\cal C}_{1}^{c}(M))\leq\varepsilon$. By applying the limiting result (21) of Lemma 4.2, $\lim_{\delta\to 0}\sup_{N\geq 1}\mathbb{Q}_{\mu}^{N}({\cal C}_{2}(\delta,n^{-1}))=0$ for each $n\geq 1$. Then, there exists $\delta_{n}>0$ satisfying $\lim_{n\to\infty}\delta_{n}=0$ such that $\sup_{N\geq 1}\mathbb{Q}_{\mu}^{N}({\cal C}_{2}^{c}(\delta_{n},n^{-1}))\leq\frac{\varepsilon}{2^{n}}$. Define ${\cal C}:={\cal C}_{1}(M)\cap(\bigcap_{n\geq 1}{\cal C}_{2}(\delta_{n},n^{-1}))\subset\hat{S}$. Then ${\cal C}$ is relatively compact in $\hat{S}$, and it follows from the above given estimates that $\sup_{N\geq 1}\mathbb{Q}_{\mu}^{N}({\cal C}^{c})\leq 2\varepsilon$. This shows that $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}$ is relatively compact in ${\cal P}(\hat{S})$.

• •

The proof of (II): $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}$ satisfies the uniform integrability condition (26).

For any $N\geq 1$, it holds that

[TABLE]

It follows from Jensen’s inequality that, for some constant $C_{\epsilon}>0$ which only depends on $\epsilon$,

[TABLE]

Observe that $\hat{\vartheta}\in\hat{S}$. Using the assumption (Aε,f,ρ), Lemma A.1 and noting that $\zeta^{i}\in\Xi_{K}$ for all $i\geq 1$, it follows from (23) and (31) that, as $R\to\infty$,

[TABLE]

i.e., the uniform integrability condition (26) holds. This completes the proof of the lemma. ∎

The following proposition completes step (ii).

Proposition 4.4**.**

Under the assumptions of Theorem 4.1 hold, $(\mathbb{Q}^{N})_{N=1}^{\infty}\subset{\cal P}({\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S})$ is tight. If the law of a ${\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S}$-valued r.v. $(\hat{\mu}_{0},\hat{\theta},\hat{\mu})$ defined on some probability space $(\hat{\Omega},\hat{\mathcal{F}},\hat{\mathbb{P}})$ is the weak limit of a convergent subsequence of $(\mathbb{Q}^{N})_{N=1}^{\infty}$, then, $\hat{\mathbb{P}}$-a.s. $\hat{\mu}$ is the unique solution of FPK equation (18) with initial condition $\hat{\mu}(0)=\hat{\mu}_{0}$.

Proof.

The tightness of $(\mathbb{Q}^{N})_{N=1}^{\infty}$ follows from the condition (17) and Lemma 4.3. Since $\hat{\mathbb{P}}\circ(\hat{\mu}_{0},\hat{\theta},\hat{\mu})^{-1}$ is the weak limit of a convergent subsequence of $(\mathbb{Q}^{N})_{N=1}^{\infty}$, Skorokhod representation theorem implies the existence of a probability space $(\Omega^{*},\mathcal{F}^{*},\mathbb{P}^{*})$, a sequence of ${\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S}$-valued r.v.s $(\mu_{0}^{N,*},\theta_{N}^{*},\mu^{N,*})$ and $(\mu^{*}_{0},\theta^{*},\mu^{*})$ satisfying $\mathbb{P}^{*}\circ(\mu_{0}^{N,*},\theta_{N}^{*},\mu^{N,*})^{-1}=\mathbb{Q}^{N}$, $\mathbb{P}^{*}\circ(\mu^{*}_{0},\theta^{*},\mu^{*})^{-1}=\hat{\mathbb{P}}\circ(\hat{\mu}(0),\hat{\theta},\hat{\mu})^{-1}$, and $\mathbb{P}^{*}$-a.s., as $N\to\infty$,

[TABLE]

Moreover, we also have that, for all $p\geq 1$,

[TABLE]

The first equality in (35) follows from the fact that $\mathbb{P}^{*}\circ(\mu_{0}^{N,*},\theta_{N}^{*},\mu^{N,*})^{-1}=\mathbb{Q}^{N}$ with $\mathbb{Q}^{N}:=Q_{N}\circ(\mu^{N}(0),\theta_{N},\mu^{N})^{-1}$. As for the second equality in (35), observe that an application of Itô’s formula yields ${\cal M}^{N}(t)=\langle\mu^{N}(t),\varphi(t)\rangle-\langle\mu^{N}(0),\varphi(0)\rangle-\int_{0}^{t}\langle\mu^{N}(s),{{\cal A}^{\theta_{N},\langle\mu^{N}(s),\rho\rangle}}\varphi(s)\rangle ds$ for any $t\in[0,T]$, where ${\cal M}^{N}(t):=\frac{1}{N}\sum_{i=1}^{N}\int_{0}^{t}\nabla_{x}\varphi(s,\xi^{i},Z_{N}^{i}(s),X_{N}^{i}(s))^{\top}\varepsilon^{i}dW_{N}^{i}(s)$. Observe that the test function $\varphi\in{\cal D}$, and $(W_{N}^{1},\ldots,W_{N}^{N})$ are independent Wiener processes under the probability measure $Q_{N}$. Then, from the BDG inequality, there exists a constant $C>0$, independent of $N$, such that

[TABLE]

This shows that the second equality in (35) must hold.

We next claim that for any $t\in[0,T]$ and test function $\varphi\in{\cal D}$, $\mathbb{P}^{*}$-a.s.

[TABLE]

where the mapping $\Upsilon_{t,\varphi}:\hat{S}\times{\cal C}_{m}\times{\cal C}_{1}\to\mathbb{R}$ is defined as:

[TABLE]

Here, the definition of ${\cal A}^{\theta,h(s)}$ is given in (16). Then, for any $(\theta_{i},h_{i})\in{\cal C}_{m}\times{\cal C}_{1}$ with $i=1,2$ and $\mu\in\hat{S}$, it follows from (Aε,f,ρ) that there exists a constant $C_{\varphi}>0$ such that

[TABLE]

Moreover, by (Aε,f,ρ), $|{{\cal A}^{\theta,h(s)}}\varphi(s,e)|\leq C_{\varphi}[1+\|\theta\|_{T}+\|h\|_{T}+|e|^{2}]$. Hence, for any $s\in[0,T]$, by (34) and Theorem 7.12 of Villani (2003), we arrive at the conclusion that, $\mathbb{P}^{*}$-a.s., $\langle\mu^{N,*}(s),{{\cal A}^{\theta,h(s)}}\varphi(s)\rangle\to\langle\mu^{*}(s),{{\cal A}^{\theta,h(s)}}\varphi(s)\rangle$ as $N\to\infty$. Further, for $q(e):=|e|^{2}$, it holds that, $\mathbb{P}^{*}$-a.s.

[TABLE]

for some positive constant $C_{\varphi,h}$ which is independent of $N$. Using the limiting results given in (34), we then obtain that $d_{\hat{S}}(\mu^{N,*},\mu^{*})\to 0$ as $N\to\infty$, $\mathbb{P}^{*}$-a.s. Note that $\mu^{*}\in\hat{S}$. Then, it follows from the dominated convergence theorem that, for $t\in[0,T]$,

[TABLE]

Using (38), (34) and (40), we deduce that, $\mathbb{P}^{*}$-a.s.

[TABLE]

This proves the limit in (36). By applying Fatou’s lemma, (35) and (36), we obtain that

[TABLE]

This proves (18) for all $\omega\in\hat{\Omega}_{0}$ with some $\hat{\Omega}_{0}\subset\hat{\Omega}$ satisfying $\hat{\mathbb{P}}(\hat{\Omega}_{0})=1$.

We next prove the uniqueness of a solution to the FPK equation (18) in the trajectory sense. This is done by verifying the conditions (DH1)-(DH4) imposed in Theorem 4.4 of Manita et al. (2015). For fixed $\omega\in\hat{\Omega}_{0}$, and for $(t,e,\mu)\in[0,T]\times E\times{\cal P}_{2}(E)$, define

[TABLE]

It then follows from (43) that $\sqrt{A(e)}$ is twice differentiable in $e$, and hence the assumption (DH1) in Theorem 4.4 of Manita et al. (2015) is satisfied. Choose the convex function $\Phi\in C^{2}(E)$ as $\Phi(e)=1+|e|^{2}$ for $e\in E$. For any $\hat{e}=(\hat{\varepsilon},\hat{y},\hat{x})\in E$, it follows from (Aε,f,ρ) that there exists a constant $C_{f,\phi}>0$ (which may vary from line to line) such that

[TABLE]

and there exists a constant $C_{f,\phi,\mu,\omega}>0$ such that

[TABLE]

It also follows from (Aε,f,ρ) that

[TABLE]

This complete the verification of the condition (DH2). For any $\mu,\nu\in{\cal P}_{2}(E)$, it follows from (43) and (Aε,f,ρ) that

[TABLE]

where $G(y):=[f]_{\rm Lip}[\rho]_{\rm Lip}y$ for $y\in[0,\infty)$. Obviously, the function $G$ is continuous and increasing on $[0,\infty)$ with $G(0)=0$. This verifies the condition (DH3). Next, we verify the condition (DH4). Take $\Psi(e)=\sqrt{\Phi(e)}$ for $e\in E$. Hence, $\Psi\in C^{2}(E)$, $\Psi\geq 1$, and $|\nabla_{e}\Psi(e)|+|\nabla_{ee}^{2}\Psi(e)|\leq C$ for some constant $C>0$. Moreover, it holds that $\lim_{|e|\to\infty}\Psi(e)=+\infty$. We deduce from (43) that, for any $\mu\in{\cal P}_{2}(E)$,

[TABLE]

For fixed $\omega\in\hat{\Omega}_{0}$, the uniqueness of a solution to (18) follows from Theorem 4.4 in Manita et al. (2015). ∎

The next lemma concludes the step (iii) outlined in the proof roadmap. Its proof follows directly from the Gluing lemma (see, e.g. Lemma 7.6 in Villani (2003)) and is omitted here.

Lemma 4.5**.**

Let assumptions of Theorem 4.1 hold. Then, the precompact sequence $(\mathbb{Q}^{N})_{N=1}^{\infty}$ has a unique weak limit point.

We now have all the ingredients to prove the main result (Theorem 4.1) of this section.

Proof of Theorem 4.1.

It follows from Proposition 4.4 and Lemma 4.5 that $(\mathbb{Q}^{N})_{N=1}^{\infty}\subset{\cal P}({\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S})$ is convergent under the weak topology. Let us endow $O:={\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S}$ with the following metric: for $o_{i}=(\vartheta_{0i},w_{i},\vartheta_{i})\in O$ with $i=1,2$,

[TABLE]

Then, using assumptions (Aε,f,ρ) and (AΘ), the fact that $\zeta_{N}=(\zeta_{N}^{i})_{i=1}^{\infty}\in\Xi_{K}^{\mathbb{N}}$, and Lemma A.1, for $\hat{o}=(\delta_{0},0,\delta_{0})\in O$ and $\epsilon>0$, it follows that

[TABLE]

where $C$ and $C_{K,\epsilon,T}$ are some positive constants independent of $N$. This implies that

[TABLE]

Then, the convergence of $(\mathbb{Q}^{N})_{N=1}^{\infty}$ in ${\cal P}_{2}(O)$ follows from Theorem 7.12 in Villani (2003) along with the uniform integrability result given in (51). ∎

Lastly, we show the uniqueness of the weak limit point of the marginal distributions $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}$ defined by (19). Lemma 4.3 shows that $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}\subset{\cal P}_{2}(\hat{S})$ is precompact. The following corollary is an immediate consequence of Theorem 4.1.

Corollary 4.6**.**

Let assumptions of Theorem 4.1 hold. Then, the precompact sequence $(\mathbb{Q}_{\mu}^{N})_{N=1}^{\infty}\subset{\cal P}_{2}(\hat{S})$ has a unique limit $\mathbb{Q}_{\mu}^{*}\in{\cal P}_{2}(\hat{S})$ satisfying ${\cal W}_{\hat{S},2}(\mathbb{Q}_{\mu}^{N},\mathbb{Q}_{\mu}^{*})\to 0$ as $N\to\infty$.

4.2 A Sufficient Condition for Weak Convergence (17)

This section provides an easily verifiable sufficient condition on the initial sample law $\nu\in{\cal P}(\Xi_{K}^{\mathbb{N}})$ that guarantees the weak convergence (17) assumed in Theorem 4.1:

• (Aν)

For $N\in\mathbb{N}$, define the mapping $I_{N}:\Xi_{K}^{\mathbb{N}}\to{\cal P}_{2}(E)$ as follows: for any $\hat{\zeta}=(\hat{X}^{i},\hat{Y}^{i})_{i=1}^{\infty}\in\Xi_{K}^{\mathbb{N}}$, $I_{N}(\hat{\zeta}):=\frac{1}{N}\sum_{i=1}^{N}\delta_{(\varepsilon^{i},\hat{Y}^{i},\hat{X}^{i})}$. Then, there exists a measurable mapping $I_{*}:\Xi_{K}^{\mathbb{N}}\to{\cal P}_{2}(E)$ such that

[TABLE]

The following remark presents an example of initial laws of training samples that satisfy (Aν):

Remark 4.7**.**

Consider any sequence of i.i.d. $\Xi_{K}$-valued r.v.s $(\hat{X}^{i},\hat{Y}^{i})_{i=1}^{\infty}$ on some probability space $(\hat{\Omega},\hat{\mathcal{F}},\hat{\mathbb{P}})$. Set $\hat{\zeta}=(\hat{\zeta}^{i})_{i=1}^{\infty}=(\hat{X}^{i},\hat{Y}^{i})_{i=1}^{\infty}$ and hence $\hat{\zeta}\in\Xi_{K}^{\mathbb{N}}$. Then, for any sequence $(\varepsilon^{i})_{i=1}^{\infty}$ satisfying $\lim_{i\to\infty}\varepsilon^{i}=\varepsilon^{*}$, the law of large of number (LLN) yields $\hat{\mathbb{P}}(\{\omega\in\hat{\Omega};\ \lim_{N\to\infty}{\cal W}_{E,2}(I_{N}(\hat{\zeta}(\omega)),I_{*})=0\})=1$. In this specific setup, $I_{*}:=\delta_{(\varepsilon^{*},\gamma^{*},\sigma^{*})}\otimes\hat{\mathbb{P}}\circ(\hat{\zeta}^{1})^{-1}$. Consider the initial sample law $\nu:=\hat{\mathbb{P}}\circ(\hat{\zeta})^{-1}$, then it holds that

[TABLE]

Hence, the assumption (Aν)* is satisfied in this specific setup.*

Note that the convergence of relaxed controls $(Q_{N})_{N=1}^{\infty}$ does not imply the weak convergence (17). The reason is that the empirical distribution of the initial data may not converge. Condition (Aν) guarantees that the distribution of initial data is well behaved. The following lemma, proven in the Appendix, shows that if the convergence of $(Q_{N})_{N=1}^{\infty}$ converges, then (Aν) implies the weak convergence in (17).

Lemma 4.8**.**

Let (Aν)* hold. Consider an arbitrary sequence $(Q_{N})_{N=1}^{\infty},Q\subset{\cal Q}(\nu)$ with $\lim_{N\to\infty}{\cal W}_{{\Omega_{\infty}},2}(Q_{N},Q)=0$, then $Q_{N}\circ(\zeta_{N},I_{N}(\zeta_{N}),\theta_{N})^{-1}=\!\!\Rightarrow Q\circ(\zeta,I_{*}(\zeta),\theta)^{-1}$, $N\to\infty$. Here, $(\zeta,W,\theta)$ (resp. $(\zeta_{N},W_{N},\theta_{N})$) is the coordinate process corresponding to $Q$ (resp. $Q_{N}$).*

4.3 Convergence of Sampled Objective Functionals

In this section, we prove the convergence, as the number of samples $N\to\infty$, of the sampled objective functional $J_{N}(Q)$ given by (8), for a fixed $Q\in{\cal Q}(\nu)$. Such an analysis uses the generalized convergence result given in Theorem 4.1.

For fixed $Q\in{\cal Q}(\nu)$, let $(\zeta,W,\theta)$ be the canonical (or coordinate) process corresponding to $Q$. For $i\geq 1$, recall that ${X}^{\theta,i}=(X^{\theta,i}(t))_{t\in[0,T]}$ solves the SDE (1) driven by $(\zeta,W,\theta)$. Next, we introduce a new empirical measure-valued process given by

[TABLE]

The empirical process $\hat{\mu}^{N}=(\hat{\mu}^{N}(t))_{t\in[0,T]}$ can be viewed as the counterpart of $\mu^{N}$ defined in (14), but driven by $(\zeta,W,\theta)$ instead of the sample $(\zeta_{N},W_{N},\theta_{N})$ from $Q_{N}\in{\cal Q}(\nu)$. We then define the law of $\hat{\mu}^{N}$ as:

[TABLE]

Using (54) and (55), we may rewrite the sampled objective functional $J_{N}(Q)$ in (8) as follows:

[TABLE]

In the above expression, we recall that $\langle\mu,f\rangle:=\int fd\mu$ for $\mu\in{\cal P}(E)$ and the loss function is defined by $L(e)=\alpha|x-y|^{2}$ where $e=(\varepsilon,y,x)\in E$. By applying Lemma 4.8 and Corollary 4.6, we immediately get the following result.

Lemma 4.9**.**

Let (Aε,f,ρ), (AΘ) and (Aν)* hold. Then, the precompact sequence $(\hat{\mathbb{Q}}^{N})_{N=1}^{\infty}\subset{\cal P}_{2}(\hat{S})$ has a unique limit point $\hat{\mathbb{Q}}^{*}\in{\cal P}_{2}(\hat{S})$ satisfying ${\cal W}_{\hat{S},2}(\hat{\mathbb{Q}}^{N},\hat{\mathbb{Q}}^{*})\to 0$ as $N\to\infty$.*

Moreover, the limit point $\hat{\mathbb{Q}}^{*}\in{\cal P}_{2}(\hat{S})$ can be explicitly characterized, as shown in the following lemma whose proof is reported in the Appendix.

Lemma 4.10**.**

Let (Aε,f,ρ), (AΘ) and (Aν)* hold. Assume $(Q_{N})_{N=1}^{\infty},Q\subset{\cal Q}(\nu)$ satisfy $\lim_{N\to\infty}{\cal W}_{\Omega_{\infty},2}(Q_{N},Q)=0$. Then $\mathbb{Q}^{N}$ defined by (15) converges to $Q\circ(I_{*},\theta,\mu_{*})^{-1}$ in ${\cal P}_{2}({\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S})$, as $N\to\infty$, where $I_{*}$ is given in (Aν)**, and $Q$-a.s., $\mu_{*}$ is the unique solution of FPK equation: for all $\varphi\in{\cal D}$,*

[TABLE]

Moreover, it holds that $\hat{\mathbb{Q}}^{*}=Q\circ(I_{*},\theta,\mu_{*})^{-1}$.

For a given $Q\in{\cal Q}(\nu)$ and the unique limit point $\hat{\mathbb{Q}}^{*}\in{\cal P}_{2}(\hat{S})$ from Lemma 4.10, we define

[TABLE]

By Lemma A.1 in the Appendix, we then have that $\sup_{N\geq 1}J_{N}(Q)<+\infty$ for each $Q\in{\cal Q}(\nu)$. We are now ready to state the main result of this section:

Proposition 4.11**.**

Let (Aε,f,ρ), (AΘ) and (Aν)* hold. Then, for any $Q\in{\cal Q}(\nu)$,*

[TABLE]

where $J_{N}(Q)$ and $J(Q)$ for $Q\in{\cal Q}(\nu)$ are defined by (56) and (60) respectively.

Proof.

To prove the proposition, we apply Lemma 4.9 and Theorem 7.12 of Villani (2003). Let $t\in[0,T]$, and define ${\cal L}^{t}(\vartheta):=\langle\vartheta(t),L\rangle=\int_{E}L(e)\vartheta(t,de)$ for all $\vartheta\in\hat{S}=C([0,T];{\cal P}_{2}(E))$. First of all, it follows from (22) that

[TABLE]

where $e=(\varepsilon,y,x)\in E$. This shows that ${\cal L}^{t}$ satisfies the growth condition on $(\hat{S},d_{\hat{S}})$ with $p=2$ in Theorem 7.12-(iv) of Villani (2003). Next, assume that $(\vartheta_{l})_{l\geq 1}\subset\hat{S}$ satisfy $\vartheta_{l}\to\vartheta$ on $(\hat{S},d_{\hat{S}})$, as $l\to\infty$. This implies that $\sup_{t\in[0,T]}{\cal W}_{E,2}(\vartheta_{l}(t),\vartheta(t))\to 0$ as $l\to\infty$. Using Theorem 7.12 of Villani (2003), it follows that for any continuous function $\phi$ on $E$ satisfying the quadratic growth, $\langle\vartheta_{l}(t),\phi\rangle\to\langle\vartheta(t),\phi\rangle$ as $l\to\infty$. Note that $\phi(e):=|x-y|^{2}\leq 2|e|^{2}$ and hence ${\cal L}^{t}(\vartheta_{l})\to{\cal L}^{t}(\vartheta)$ as $l\to\infty$. Thus, we have shown that ${\cal L}^{t}$ is continuous and satisfies the quadratic growth on $(\hat{S},d_{\hat{S}})$. By Lemma 4.9, ${\cal W}_{\hat{S},2}(\hat{\mathbb{Q}}^{N},\hat{\mathbb{Q}}^{*})\to 0$ as $N\to\infty$. Again, by Theorem 7.12 of Villani (2003), we conclude that

[TABLE]

Moreover, for $\epsilon>0$, using Jensen’s inequality and Lemma A.1, we deduce the existence of a positive constant $C_{\epsilon,T}$ which only depends on $\epsilon,T$ such that

[TABLE]

This implies that, as $R\to\infty$,

[TABLE]

Then, by Vitali’s convergence theorem together with (63) and (4.3), it follows that

[TABLE]

The desired convergence then follows from (63) and (66), recalling the expressions of $J_{N}(Q)$ and $J(Q)$ given, respectively, by (56) and (60). ∎

5 Convergence of Minimizers of Sampled Objective Functionals

In this section, we show that the sequence of minimizers of the sampled objective functionals converges to the minimizer of the limiting objective functional, if $N$ is large enough. To establish this result, a key step is to prove the so-called $\Gamma$-convergence of $J_{N}$ to $J$ (see (56) and (60)).

Before introducing the main result of this section, we first metrize the space ${\cal Q}(\nu)\subset{\cal P}_{2}(\Omega_{\infty})$ by taking the quadratic Wasserstein distance ${\cal W}_{\Omega_{\infty},2}$ on ${\cal Q}(\nu)$. The main result of the paper is as follows:

Theorem 5.1**.**

Let (Aε,f,ρ), (AΘ) and (Aν)* hold. Then, it holds that*

[TABLE]

where the minimum of $J(Q)$ over $Q\in{\cal Q}(\nu)$ exists. Moreover, if the minimizing sequence $(Q_{N})_{N=1}^{\infty}\subset{\cal Q}(\nu)$ (up to a subsequence) converges to some $Q^{*}\in{\cal Q}(\nu)$ in ${\cal W}_{\Omega_{\infty},2}$, then $Q^{*}$ minimizes $J(Q)$ over $Q\in{\cal Q}(\nu)$.

Proof.

The proof of Theorem 5.1 requires proving (i) the Gamma-convergence of $J_{N}$ to $J$, which is done in Proposition 5.2; and (ii) that the minimizing sequence of the sampled optimization problem (9) is precompact in ${\cal W}_{\Omega_{\infty},2}$, which is shown in Lemma 5.3. ∎

We next give the definition of Gamma-convergence of the sequence of sampled objective functionals $(J_{N})_{N=1}^{\infty}$ on $({\cal Q}(\nu),{\cal W}_{\Omega_{\infty},2})$ (see, e.g. Dal Maso (1993)):

Definition 5.1**.**

$J_{N}:{\cal Q}(\nu)\to\mathbb{R}$* Gamma-converges to some functional $J:{\cal Q}(\nu)\to\mathbb{R}$, i.e., $J=\Gamma\mbox{-}\lim_{N\to\infty}J_{N}$ on ${\cal Q}(\nu)$, if the following conditions hold:*

• (i)

(liminf inequality):* For any $Q\in{\cal Q}(\nu)$ and every sequence $(Q_{N})_{N=1}^{\infty}$ converging to $Q$ in $({\cal Q}(\nu),{\cal W}_{\Omega_{\infty},2})$, we have that $\liminf_{N\to\infty}J_{N}(Q_{N})\geq J(Q)$;*

• (ii)

(limsup inequality):* For any $Q\in{\cal Q}(\nu)$, there exists a sequence $(\bar{Q}_{N})_{N=1}^{\infty}$ which converges to $Q$ in $({\cal Q}(\nu),{\cal W}_{\Omega_{\infty},2})$ (this sequence is said to be a $\Gamma$-realising sequence), such that $\limsup_{N\to\infty}J_{N}(\bar{Q}_{N})\leq J(Q)$.*

The following proposition shows that $J_{N}$ Gamma-converges to $J$ as $N\to\infty$.

Proposition 5.2**.**

Let (Aε,f,ρ), (AΘ) and (Aν)* hold. Then $J=\Gamma\mbox{-}\lim_{N\to\infty}J_{N}$ on $({\cal Q}(\nu),{\cal W}_{\Omega_{\infty},2})$.*

Proof.

Let $Q\in\mathcal{Q}(\nu)$ and take $\bar{Q}_{N}=Q$ for all $N\geq 1$. Then, it follows from Proposition 4.11 that $\lim_{N\to\infty}J_{N}(\bar{Q}_{N})=J(Q)$. Therefore, $(\bar{Q}_{N})_{N=1}^{\infty}\subset{\cal Q}(\nu)$ is a $\Gamma$-realising sequence. Hence, the $\limsup$ inequality in Definition 5.1 holds.

It remains to prove the $\liminf$ inequality. Let $(Q_{N})_{N=1}^{\infty},Q\subset{\cal Q}(\nu)$ satisfy $\lim_{N\to\infty}{{\cal W}_{\Omega_{\infty},2}}(Q_{N},Q)=0$. Then, it follows from Lemma 4.10 that $\mathbb{Q}^{N}=Q_{N}\circ(\mu^{(N)}(0),\theta_{N},\mu^{N})^{-1}$ converges to $Q\circ(I_{*},\theta,\mu_{*})^{-1}$ in ${\cal P}_{2}({\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S})$. The exact expression of $(I_{*},\mu_{*})$ is given in Lemma 4.10. Recall the expression of ${\mathbb{Q}}^{N}_{\mu}$ given in (19). Then ${\mathbb{Q}}^{N}_{\mu}$ converges to $\hat{\mathbb{Q}}^{*}:=Q\circ\mu_{*}^{-1}$ in ${\cal P}_{2}(\hat{S})$, as $N\to\infty$. Using similar arguments to those in the proof of Proposition 4.11, this leads to

[TABLE]

We next state and prove the following claim:

[TABLE]

If $\ell=+\infty$, then (69) trivially holds. If $\ell<+\infty$ (note that $\ell\geq 0$ by the way it is defined), then passing to a subsequence (call it $Q_{N}$ again), we may assume that

[TABLE]

Note that $Q_{N}\circ\theta_{N}^{-1}\Rightarrow Q\circ\theta^{-1}$ as $N\to\infty$. Using Skorokhod’s representation theorem, there exists a probability space $(\Omega^{*},\mathcal{F}^{*},\mathbb{P}^{*})$, a sequence of ${\cal C}_{m}$-valued r.v.s $(\theta_{N}^{*})_{N=1}^{\infty}$, $\theta^{*}$ such that $\mathbb{P}^{*}\circ(\theta_{N}^{*})^{-1}=Q\circ\theta_{N}^{-1}$, $\mathbb{P}^{*}\circ(\theta^{*})^{-1}=Q\circ\theta^{-1}$, and as $N\to\infty$, $\theta_{N}^{*}\to\theta^{*}$ in ${\cal C}_{m}$, $\mathbb{P}^{*}$-a.s. It follows from the dominated convergence theorem that

[TABLE]

We next prove that $({\theta_{N}^{*}}^{\prime})_{N=1}^{\infty}$ is bounded in $L^{2}((0,T)\times\Omega^{*};dt\otimes d\mathbb{P}^{*})$. This can be derived using similar arguments to those employed to derive (A.22). For completeness, we provide the precise mathematical details. For any $\phi\in{\cal D}$, define ${\cal T}_{N}(\phi):=(\theta_{N}^{*},\phi^{\prime})$. Let $(\phi_{l})_{l=1}^{\infty}\subset{\cal D}$ be dense in ${\cal L}_{m}^{2}$. Then, for each $N\geq 1$,

[TABLE]

By Hahn-Banach theorem and Riesz representation theorem, there exists an ${\cal L}_{m}^{2}$-valued r.v. $\hat{\theta}_{N}^{*}$ such that ${\cal T}_{N}(\phi)=(\hat{\theta}_{N}^{*},\phi)$ for all $\phi\in{\cal L}_{m}^{2}$, $\mathbb{P}^{*}$-a.s. In particular, ${\cal T}_{N}(\phi)=(\theta_{N}^{*},\phi^{\prime})=(\hat{\theta}_{N}^{*},\phi)$ for any $\phi\in{\cal D}$. This yields that $\theta_{N}^{*}\in{\cal H}_{m}^{1}$, $\mathbb{P}^{*}$-a.s. It then follows from (70) that

[TABLE]

This shows that $({\theta_{N}^{*}}^{\prime})_{N=1}^{\infty}$ is bounded in $L^{2}((0,T)\times\Omega^{*};dt\otimes d\mathbb{P}^{*})$, and hence ${\theta_{N}^{*}}^{\prime}$ (up to a subsequence) converges weakly to some $\hat{\theta}^{*}\in L^{2}((0,T)\times\Omega^{*};dt\otimes d\mathbb{P}^{*})$ as $N\to\infty$. As shown in Proposition 3.1, for any $\phi\in{\cal D}$ and $H\in L^{\infty}(\Omega^{*};\mathbb{P}^{*})$, by the weak convergence property

[TABLE]

Thus, for any $\phi\in{\cal D}$, $(\theta^{*},\phi^{\prime})=-(\hat{\theta}^{*},\phi)$, $\mathbb{P}^{*}$-a.s.. The separability of ${\cal D}$ implies that $\mathbb{P}^{*}$-a.s., $(\theta^{*},\phi^{\prime})=-(\hat{\theta}^{*},\phi)$, for all $\phi\in{\cal D}$, i.e., ${\theta^{*}}^{\prime}=\hat{\theta}^{*}$, $\mathbb{P}^{*}$-a.s.. Similarly to the derivation of the estimates (A.22) and (73), we obtain that

[TABLE]

Thus, the proof of (69) follows immediately from (71) and (75). Finally, note that

[TABLE]

Then, the $\liminf$ inequality in Definition 5.1 follows from (60), (68), and (69). ∎

To conclude the proof of Theorem 5.1, it remains to establish the relatively compactness of the minimizing sequence of the sampled optimization problem (9). As shown in Proposition 3.1 of Section 3, for each $N\geq 1$ there exists a relaxed solution $Q_{N}\in{\cal Q}(\nu)$ such that $J_{N}(Q_{N})=\inf_{Q\in{\cal Q}(\nu)}J_{N}(Q)$. We will prove below that such a sequence $(Q_{N})_{N=1}^{\infty}$ is precompact under the quadratic Wasserstein distance ${\cal W}_{\Omega_{\infty},2}$.

Lemma 5.3**.**

Let (Aε,f,ρ)* and (AΘ)** hold. Then, the above minimizing sequence $(Q_{N})_{N=1}^{\infty}\subset{\cal Q}(\nu)$ is precompact in ${\cal W}_{\Omega_{\infty},2}$.*

Proof.

Lemma A.1 implies that $\sup_{N\geq 1}J_{N}(Q)\leq\ell$ for some $\ell>0$. Together with Proposition 3.1, this implies the existence of a constant $\ell>0$ and some $Q\in{\cal Q}(\nu)$ such that

[TABLE]

Therefore, it follows that

[TABLE]

Let ${\cal X}_{N}=(\zeta_{N},W_{N},\theta_{N})$ be the canonical process corresponding to $Q_{N}$. Using similar arguments to those in the proof of Proposition 3.1, $(Q_{N}\circ\theta^{-1}_{N})_{N=1}^{\infty}$ is tight on ${\cal C}_{m}$, and moreover $(Q_{N}\circ{\cal X}^{-1}_{N})_{N=1}^{\infty}$ is tight on $\Omega_{\infty}$ because $(\Omega_{\infty},d)$ is Polish. Then, by Prokhov’s theorem, there exists a $Q^{*}\in{\cal P}(\Omega_{\infty})$ such that the minimizing sequence $(Q_{N})_{N=1}^{\infty}$, up to a subsequence, converges to $Q^{*}$ under weak topology. Using Skorokhod representation theorem, there exists a probability space $(\Omega^{*},\mathcal{F}^{*},\mathbb{P}^{*})$, ${\cal X}_{N}^{*}=(\zeta^{*}_{N},W_{N}^{*},\theta_{N}^{*})$ with ${\cal X}_{N}^{*}\overset{d}{=}{\cal X}_{N}$, and ${\cal X}^{*}=(\zeta^{*},W^{*},\theta^{*})$ with $\mathbb{P}^{*}\circ({\cal X}^{*})^{-1}=Q^{*}$ such that, $\mathbb{P}^{*}$-a.s., as $N\to\infty$,

[TABLE]

Further, we obtain that (i): for $N\geq 1$, $\theta_{N}^{*}\in{\cal H}_{m}^{1}$ $\mathbb{P}^{*}$-a.s. and hence $\mathbb{P}^{*}\circ({\cal X}_{N}^{*})^{-1}\in{\cal P}(\Omega_{\infty})$; (ii): $Q^{*}=\mathbb{P}^{*}\circ({\cal X}^{*})^{-1}\in{\cal P}(\Omega_{\infty})$, and hence $Q^{*}\in{\cal Q}(\nu)$. To prove that $Q_{N}$ converges to $Q^{*}$ as $N\to\infty$ in ${\cal W}_{\Omega_{\infty},2}$, using Definition 6.8 and Theorem 6.9 in Villani (2009), it suffices to prove that

[TABLE]

for some $(\hat{\gamma},\hat{w},\hat{\vartheta})\in\Omega_{\infty}$. First, note that $Q_{N}=\mathbb{P}^{*}\circ({\cal X}_{N}^{*})^{-1}$. Then, for all $N\geq 1$, it holds that

[TABLE]

Observing that, for any $N\geq 1$, $(\zeta^{*}_{N},W^{*}_{N})$ and $(\zeta^{*},W^{*})$ are identically distributed by (i) and (ii) of Definition 3.1, it follows that $(d^{2}_{1}(\zeta_{N}^{*},\hat{\gamma})+d^{2}_{2}(W_{N}^{*},\hat{w}))_{N=1}^{\infty}$ is uniformly integrable. Using (AΘ), we deduce that $(d^{2}_{3}(\theta_{N}^{*},\hat{\vartheta}))_{N=1}^{\infty}$, is uniformly integrable. It then follows from (79) and Vitali’s convergence theorem that

[TABLE]

which yields (80). This completes the proof of the lemma. ∎

Remark 5.4**.**

If $\lambda_{2}=0$ in (6), one can apply a similar argument to the one used in the case $\lambda_{2}>0$ to characterize the limiting objective $J(Q)=\lim_{N\to\infty}J_{N}(Q)$ and the Gamma convergence $J_{N}\overset{\Gamma}{\to}J$ as $N\to\infty$. However, without derivative-based regularization, one needs a different argument to esablish the convergence of minimizers as the sample size grows to infinity. See Appendix A for more details, where we require the forcing function $f$ specified in Section 7 to satisfy the condition (89) .

6 Connection to Deep ResNet

In this section, we demonstrate how the proposed framework can mimic the training process of ResNets. Section 6.1 illustrates how a stylized unregularized ResNet with infinite depth can be modeled by a continuous time deterministic dynamical system. Section 6.2 shows how, extending the deterministic to a stochastic system of the form in Eq. (1), one can model regularization during the training process. Section 6.3 discusses how the optimization criterion considered in the previous sections is related to regularized loss functions of neural networks.

6.1 Representation of Deep ResNet as a Dynamical System

We begin by considering a simplified version of the $n$-layered ResNet architecture, as in Thorpe and van Gennip (2020). For $l=0,1,\ldots,n-1$, let $d$ be the number of neurons in each layer, and $X^{(n)}(l)\in\mathbb{R}^{d}$ the states of neurons in layer $l$. We use $w^{(n)}(l)\in\mathbb{R}^{d\times d}$ to denote the matrix of weights which determine how neurons in layer $l$ activate neurons in layer $l+1$, and $b^{(n)}(l)\in\mathbb{R}^{d}$ to denote the vector of biases at layer $l$. The feed-forward propagation in the $n$-layer ResNet model can be represented by the difference equation:

[TABLE]

where $\frac{1}{n}$ is the scaling factor, and $\sigma(l,x)=(\sigma_{1}(l,x_{1}),\ldots,\sigma_{d}(l,x_{d}))^{\top}$ is the activation function which effectively turns neurons “on” or “off” at layer $l+1$ based on the value of the input $x=(x_{1},\ldots,x_{d})$ at layer $l$. In existing implementations, the activation function is chosen to be a smooth approximation of a rectified linear activation function or a sigmoid function (see also Haykin (2009)). Eq. (83) highlights the residual property of the network: the endogenous input at layer $l+1$ consists of the non-transformed input $X^{(n)}(l)$ from layer $l$, plus a nonlinear transformation of $X^{(n)}(l)$. The term $X^{(n)}(l)$ represents information from the previous layer “skipping the processing associated with the layer $l$”, and being transmitted to layer $l+1$ without being transformed. Let $\theta^{(n)}(l):=(w^{(n)}(l),b^{(n)}(l))\in\mathbb{R}^{d\times d}\times\mathbb{R}^{d}$, and write

[TABLE]

As observed in prior studies (e.g. E (2017) and Lu et al. (2018)), one can turn the setting described above into an explicit Euler characterization of the ODE, yielding

[TABLE]

where $X^{\theta}(t)\in\mathbb{R}^{d}$ is the state at layer $t$ of the feed-forward transformation triggered by the the training inputs, $t\in[0,T]$ represents the depth of the network, and the time step $dt$ corresponds to the scaling factor $\frac{1}{n}$. The initial state $X^{\theta}(0)=X(0)$ collects the training inputs. Above, we have used the superscript $\theta$ to emphasize the dependence of output neurons on the weight matrix and, for a finite $T$, $X^{\theta}$ and $\theta$ denote real valued functions on $[0,T]$. The above argument has been used to justify neural network architectures arising from discretization of differential equations (e.g. Chen et al. (2018) and Thorpe and van Gennip (2020)). We will treat (85) as a differential equation, i.e., assume an infinitesimal time step. This effectively corresponds to the limit of an infinitely deep residual network, i.e., consisting of infinitely many layers.

6.2 Regularized Deep ResNet

We extend the dynamical system representation of a deep ResNet sketched in the previous section to incorporate regularization during the training process. Specifically, we allow for (i) additive noise injection into the deterministic hidden units to prevent overfitting, and (ii) batch normalization in each layer to accelerate convergence of the training algorithm. This turns the continuous time dynamics given by (85) into a stochastic dynamical system of the form in Eq. (1).

We next illustrate how the proposed framework can be used to describe continuous layer idealizations of deep ResNets at different levels of complexity.

• •

The vector function $f$ in Eq. (1) depends only on the input variables $(t,\theta,x)$, and $\varepsilon^{i}=0$. This yields a feed forward propagation in a deep network, where the randomness only comes from the initial input $X^{i}(0)$. Such a dynamical system specification recovers the framework of E et al. (2018), which has also been specified in Eq. (85).

• •

The vector function $f$ depends on all its inputs $(t,\theta,x,\eta)$, and $\varepsilon^{i}=0$. We next illustrate how this specification incorporates batch normalization (see also Algorithm 1 in Ioffe and Szegedy (2015)) through the batch function $\rho:\mathbb{R}^{d}\to\mathbb{R}^{2}$. For the purpose of illustration, we set $d=1$. The batch normalization transform applied to $X^{\theta,i}(t)$ over a batch of size $N$ is given by

[TABLE]

where $\gamma,\upsilon\in\mathbb{R}$ are, respectively, the scale and shift factors and $\iota>0$ is a constant added to the batch variance to avoid numerical instability. Take the (vector) batch function $\rho(x)=(\rho_{1}(x),\rho_{2}(x))=(x,x^{2})$. Then, we may rewrite the above batch normalization as:

[TABLE]

Because of such normalization, the state dynamics of the neural network becomes of the mean-field type. For $(t,\theta,x,\eta)=(t,(w,b),x,(\eta_{1},\eta_{2}))\in[0,T]\times\mathbb{R}^{2}\times\mathbb{R}\times\mathbb{R}^{2}$, this yields

[TABLE]

where $x\to S(t,x)=\frac{1}{1+e^{-x}}$ is the logistic activation function at layer $t$. Observe that, differently from E et al. (2018), the probability law of the state process $(X^{\theta,1},\ldots,X^{\theta,N})$ enters explicitly into the state forward dynamics.

Remark 6.1**.**

The batch function $\rho_{2}(x)=|x|^{2}$ introduced in Algorithm 1 of Ioffe and Szegedy (2015) is only locally Lipschitz. For practical matters, one can well approximate $\rho_{2}$ with the truncation function $\rho_{2}^{(M)}(x):=x^{2}{\bf 1}_{\{|x|\leq M\}}+M^{2}{\bf 1}_{\{|x|>M\}}$ where $M>0$ is sufficiently large. We can then consider an approximating system to (1), in which $\rho_{2}$ is replaced by $\rho_{2}^{(M)}$. Such a system is well-posed. Denote by $X^{\theta,i,M}=(X^{\theta,i,M}(t))_{t\in[0,T]}$ the solution of this approximating system. First, in terms of (88), for any $\theta\in\Theta$ (a compact subset of $\mathbb{R}^{m}$) and $x^{i}=(x_{1}^{i},\ldots,x_{d}^{i})\in\mathbb{R}^{d}$ ($i=1,\ldots,N$), we have that $a_{i}(t,(x^{1},\ldots,x^{N})):=f(t,\theta,x^{i},(\frac{1}{N}\sum_{j=1}^{N}\rho_{1}(x^{j}),\frac{1}{N}\sum_{j=1}^{N}\rho_{2}(x^{j})))$ satisfies the linear growth condition uniformly in $t\in[0,T]$. Since the locally Lipschitz continuity implies strong uniqueness, we have $X^{\theta,i}(t)=X^{\theta,i,M}(t)$ for $t\in[0,\tau_{M}^{i}]$ a.s., where $\tau^{i}_{M}:=\inf\{t\geq 0;\leavevmode\nobreak\ |X^{\theta,i}(t)|\geq M\}$. Because $a_{i}(t,\cdot)$ has linear growth for any $t\in[0,T]$, it follows from the Gronwall’s inequality that $\mathbb{E}[\sup_{t\in[0,T]}|X^{\theta,i}(t)|]<+\infty$. Then, for any $t\in[0,T]$, $\lim_{M\to+\infty}\mathbb{P}(\tau^{i}_{M}\leq t)=\lim_{M\to+\infty}\mathbb{P}(\sup_{s\in[0,t]}|X^{\theta,i}(s)|\geq M)\leq\lim_{M\to+\infty}\frac{1}{M}\mathbb{E}[\sup_{s\in[0,T]}|X^{\theta,i}(s)|]=0$. This implies that, for any $t\in[0,T]$, $\mathbb{P}(\bigcap_{N=1}^{+\infty}\bigcup_{M=N}^{+\infty}\{\tau^{i}_{M}\leq t\})=0$. Since $\tau_{M}^{i}$ is increasing in $M>0$, we deduce that $\lim_{M\to+\infty}\tau^{i}_{M}=+\infty$, a.s.. Thus $|X^{\theta,i,M}(t)-X^{\theta,i}(t)|\to 0$, as $M\to\infty$, a.s.. Hence, $\lim_{M\to+\infty}X^{\theta,i,M}(t)$ for $t\in[0,T]$ is the unique solution of (1).

• •

The vector function $f$ depends on all its inputs $(t,\theta,x,\eta)$, and $\varepsilon^{i}\neq 0$. This is the most general setting, in which a noise process modeled through the Brownian motion $W(t)=(W^{1}(t),\ldots,W^{N}(t))_{t\in[0,T]}$ is added along the trajectory of the deep neural network. The weight function $\theta=(\theta(t))_{t\in[0,T]}$ then becomes a $\mathbb{F}$-adapted control strategy, which will be chosen to minimize the neural network loss criterion. One can thus think of an arbitrary layer as receiving samples from a distribution that is determined by the layer below. Such a distribution changes during the course of training, making any layer except the first one responsible not only for learning a good representation, but also for adapting to a changing input distribution.

We sample noise from a zero-mean Gaussian distribution so that the noise transition function is unbiased. This means that, conditional on the input $X^{\theta,i}(t)$, the output $X^{\theta,i}(t+dt)$ at layer $t+dt$ coincides on average with the output of a deterministic ResNet. In other words, the noise-injected transition function preserves, on average, the transition function of the underlying deterministic network. This type of unbiased regularization has been shown to perform favorably compared to classical regularization techniques such as dropout, in the context of recurrent neural networks (see Dieng et al. (2018)). One can also regard the noise injected outputs of hidden units as stochastic activations, or equivalently, random selections of hidden units in a layer, consistently with Noh et al. (2017). A related study by He et al. (2019) considers noise injection as a way to make deep ResNets robust against adversarial attacks. They propose Parametric-Noise-Injection (PNI), i.e., they inject noise to different components or locations within the network.

Remark 6.2**.**

It is possible to account for a multiplicative noise injection regularization procedure. Such regularization technique, commonly referred to as the dropout, has been successfully used to mitigate the overfitting problem of overparameterized networks (see, e.g. Srivastava et al. (2014) and Ba and Frey (2013)). In our framework, this means replacing the term $\varepsilon^{i}dW^{i}(t)$ in (1) with $\varepsilon^{i}\sigma(t,\theta,X^{\theta,i}(t))dW^{i}(t)$, where $\sigma(t,\theta,x):[0,T]\times\Theta\times\mathbb{R}^{d}\to\mathbb{R}_{+}$ is the common multiplicative noise scale function. The main results developed in previous sections still hold if the following standard assumption on $\sigma$ is imposed:

• (Aσ)

1. (i)

$\sigma(t,\theta,x)$* is continuous in $(t,\theta,x)$ and is twice continuously differentiable in $x$;* 2. (ii)

$\sigma$* satisfies the following Lipschitz condition: for all $\theta_{1},\theta_{2}\in\Theta$ and $x,y\in\mathbb{R}^{d}$,*

[TABLE]

where $[\sigma]_{{\rm Lip}}$ is the Lipschitz coefficient of $\sigma$, which is independent of the layer index $t$.

We provide the corresponding proofs and supporting arguments in the Online Appendix of this paper.

6.3 Loss Function and Regularization

We can view the samples $(\zeta^{i})_{i=1}^{N}$ in Eq. (4) as $N$ i.i.d. training samples of a neural network. The $i$-th training sample consists of the training input $X^{i}(0)$, and the label part $Y^{i}(0)$.

In a neural network, the loss function quantifies the difference between the expected outcome $Y^{i}(0)$ and the endogenous output $X^{\theta,i}(T)$ produced by the learning model, for all training samples $\zeta^{i}$, $i=1,\ldots,N$. A simple and commonly used loss function is the squared-error loss specified by Eq. (6), which makes it computationally tractable to derive the gradients used to update the weights. It is also common to consider a regularizer in the objective function to penalize for model complexity. The specification in Eq. (6) is flexible enough to allow for two types of regularization. The $L^{2}$-regularization is achieved by setting $\lambda_{2}=0$, and encourages the weight values towards zero. The $L^{1}$-regularization is instead attained with $\lambda_{1}=0$, and encourages the weight values to be zero. Intuitively, smaller weights reduce the impact of hidden neurons, so that they become neglectable and the overall complexity of the neural network gets reduced.

Regularizers in the objective function of the neural network have been extensively studied. Hasan and Roy-Chowdhury (2015) consider the case where $\lambda_{1}>0$ and $\lambda_{2}=\beta=0$. If $\lambda_{1}=\lambda_{2}>0$, the control problem (2) includes an ${\cal H}_{m}^{1}$-regularizer, as in the studies of Haber and Ruthotto (2018), Thorpe and van Gennip (2020), and Oberman and Calder (2018) for the training of deep neural networks. For instance, Haber and Ruthotto (2018) consider a smoothness regularization on the weights $w$ given by $R(w)=\frac{1}{2}\|{\mathcal{L}}{\bf w}\|^{2}$, where ${\mathcal{L}}$ is a discretized differential operator. In our continuous time framework, this regularization can be understood as a discrete version of the time derivative of weights. They show that this regularization biases the learning process towards smooth time dynamics, and improves generalization of a ResNet. Oberman and Calder (2018) consider a modified loss function with Lipschitz regularization. They show theoretically that the regularized model generalizes, and empirically that the regularization improves adversarial robustness for deep neural networks (DNN).

7 Concluding Remarks and Future Directions

We have studied a class of optimal sampled (relaxed) control problems of the mean-field type, and then demonstrated how they can be used to model the optimal training of a stlyzed continuous layer deep ResNet. Building upon the existence of optimal relaxed controls for training sets of finite size, we have proven that the minimizer of the sampled (relaxed) problem converges to that of the limiting optimization problem, as the number of samples grows large.

Our work can be extended along several directions. One avenue of investigation is to study how the input dimension influences the convergence rate and the time discretization error. For mean field models without control, Luçon and Stannat (2014) study the speed at which the empirical measure of a class of disordered diffusions with singular interactions converges to the solution of a deterministic McKean-Vlasov equation. Bencheikh and Jourdain (2021) establish the rate of convergence for a system of interacting particles with mean-field rank-based interaction in the drift coefficient and constant diffusion coefficient. The core notion is the propagation of chaos for large systems of interacting particles. In the discrete time case, Motte and Pham (2022) show the role of relaxed controls in the study of Markov decision processes (MDP) under mean field interaction with common noise, and establish a rate of convergence for the problem of $N$-cooperative agents as $N$ grows to infinity. Their results can be leveraged to explore the speed of convergence of $J_{N}$ to $J$ and ${\rm argmin}\leavevmode\nobreak\ J_{N}$ to ${\rm argmin}\leavevmode\nobreak\ J$, as $N\to\infty$.

In the absence of derivative-based regularization (i.e., $\lambda_{2}=0$ in (6)), our results hold for the following linear type of forcing function:

[TABLE]

where $\theta^{(1)}\in\mathbb{R}^{d\times l}$, $\theta^{(2)}\in\mathbb{R}^{d}$. The function $S(t,x,\eta):[0,T]\times\mathbb{R}^{d}\times\mathbb{R}^{q}\to\mathbb{R}^{l}$ is continuous in $(t,x,\eta)$, and it satisfies the inequality $\|S\|_{\infty}+\|\nabla_{x}S\|_{\infty}+\|\nabla_{\eta}S\|_{\infty}<+\infty$, where $\|\cdot\|_{\infty}$ denotes the super-norm.

Our limiting results have implications for the analysis of deep ResNets. They imply that the sampled control problems arising in the optimal training of a deep ResNet can be cast as a one-dimensional stochastic control problem. Despite the solvability of the resulting one-dimensional stochastic control problem is nontrivial, this reduction allows to bypass the complexity of a large sample size during the supervised learning process. In a future continuation of this work, it would be interesting to establish a connection between the limiting stochastic control problem $\inf_{Q\in{\cal Q}(\nu)}J(Q)$ (c.f. Theorem 5.1) and a class of deterministic FPK control problems described as follows: for some $\mu_{0}\in{\cal P}_{2}(E)$,

[TABLE]

subject to the constraint:

[TABLE]

For $(\mu,\theta)\in\hat{S}\times{\cal H}_{m}^{1}$, the objective functional $J^{d}$ in (90) is given by

[TABLE]

The deterministic optimal control in (90) is obtained by minimizing the objective function under the FPK equation constraint. This type of control problems has also been studied numerically using receding-horizon control techniques, see Annunziato and Borzì (2010).

Acknowledgements

We thank the two anonymous reviewers for their careful reading of our manuscript and their insightful comments and suggestions, which certainly helped to improve our manuscript. L. Bo was supported by the Natural Science Foundation of China (Grant 11971368). A. Capponi was supported in part by the Natural Science Foundation (Grant DMS-1716145).

Appendix A Appendix

This Appendix provides proofs to some of the propositions and lemmas stated in the main body. Additionally, it provides a miscellaneous of technical results, along with the corresponding proofs, which are used to derive proofs of propositions and theorems stated in the main body of the paper.

Proof of Proposition 3.1.

Without loss of generality, we assume that $\alpha_{N}=\inf_{Q\in{\cal Q}(\nu)}J_{N}(Q)<+\infty$. In view of (8), let $(Q_{k})_{k=1}^{\infty}\subset{\cal Q}(\nu)$ be a minimizing sequence such that

[TABLE]

Above, for $k\geq 1$, the objective functional

[TABLE]

where ${\cal X}_{k}:=(\zeta_{k},W_{k},\theta_{k})=((\zeta_{k}^{i})_{i=1}^{\infty},(W_{k}^{i})_{i=1}^{\infty},\theta_{k})$ is the canonical process. Since $Q_{k}\in{\cal Q}(\nu)$, it follows from Definition 3.1 that (i) $Q_{k}\circ\zeta_{k}^{-1}=\nu$; (ii) $W_{k}$ consists of a sequence of independent Wiener processes on $(\Omega_{\infty},\mathbb{F},Q_{k})$; (iii) $\theta_{k}\in\mathbb{U}^{Q_{k},\mathbb{F}}$. Moreover, for $i=1,\ldots,N$, the state process $X_{k}^{\theta,i}$ is the strong solution of (1) driven by $(\zeta_{k},W_{k},\theta_{k})$. Then, it follows from (5), (A.1) and (A.2) that, for all $k\geq 1$,

[TABLE]

This implies that

[TABLE]

Note that $\theta_{k}$ is a ${\cal H}_{m}^{1}$ (as a subset of ${\cal C}_{m}$)-valued random variable on $(\Omega_{\infty},\mathcal{F}_{\infty},Q_{k})$ for $k\geq 1$. Then, for any $\delta>0$, it follows from Hölder inequality that

[TABLE]

For any $\epsilon>0$ and $\delta>0$, using Chebychev’s inequality, we arrive at

[TABLE]

This implies that, for any $\epsilon>0$,

[TABLE]

Using similar estimates as (A.5), we obtain that, for all $s\in[0,T]$,

[TABLE]

Integrating both sides of the above equation w.r.t. $s$, it follows from Fubini’s theorem that

[TABLE]

Then, for any $M>0$, we obtain from (A.4) that

[TABLE]

By virtue of Arzelà-Ascoli theorem (see, e.g. Simon (1987)) together with (A.7) and (A.10), we obtain that $(Q_{k}\circ\theta_{k}^{-1})_{k=1}^{\infty}$, viewed as a sequence of probability measures in ${\cal P}({\cal C}_{m})$, is tight.

Note that $\Omega_{\infty}\subset\hat{\Omega}_{\infty}$ where $\hat{\Omega}_{\infty}:=\Xi_{K}^{\mathbb{N}}\times{\cal C}_{p}^{\mathbb{N}}\times{\cal C}_{m}$. We next claim that the sequence of probability measures $(Q_{k}\circ{\cal X}_{k}^{-1})_{k=1}^{\infty}$ in ${\cal P}(\hat{\Omega}_{\infty})$ is tight. Since we have proven above that $(Q_{k}\circ\theta_{k}^{-1})_{k=1}^{\infty}\subset{\cal P}({\cal C}_{m})$, it suffices to show that $(Q_{k}\circ\zeta_{k}^{-1})_{k=1}^{\infty}\subset{\cal P}(\Xi_{K}^{\mathbb{N}})$ and $(Q_{k}\circ W_{k}^{-1})_{k=1}^{\infty}\subset{\cal P}({\cal C}_{p}^{\mathbb{N}})$ are respectively tight. Note that $(Q_{k})_{k=1}^{\infty}\subset{\cal Q}(\nu)$, then $Q_{k}\circ\zeta_{k}^{-1}=\nu$ for $k\geq 1$. Hence, the tightness of $(Q_{k}\circ\zeta_{k}^{-1})_{k=1}^{\infty}$ follows from the fact that $(\Xi_{K}^{\mathbb{N}},d_{1})$ is Polish. Similarly, it follows from (ii) of Definition 3.1 that the r.v.s $W_{k}$, for $k\geq 1$, have the same distribution. Thus, $Q_{1}\circ W_{1}^{-1}=Q_{k}\circ{W}_{k}^{-1}$ for all $k\geq 1$, and hence $(Q_{k}\circ W_{k}^{-1})_{k=1}^{\infty}\subset{\cal P}({\cal C}_{p}^{\mathbb{N}})$ is tight, because $({\cal C}_{p}^{\mathbb{N}},d_{2})$ is Polish. By Prokhorov’s theorem, there exists a $Q^{*}\in{\cal Q}(\nu)$ such that $Q_{k}\circ{\cal X}_{k}^{-1}$ converges to (up to a subsequence) $Q^{*}$ in the weak topology of probability measures. Using the Skorokhod’s representation theorem, there exists a probability space $(\Omega^{*},\mathcal{F}^{*},\mathbb{P}^{*})$, $\Omega_{\infty}$-valued r.v.s ${\cal X}_{k}^{*}=(\zeta^{*}_{k},W_{k}^{*},\theta_{k}^{*})$ with ${\cal X}_{k}^{*}\overset{d}{=}{\cal X}_{k}$, and ${\cal X}^{*}=(\zeta^{*},W^{*},\theta^{*})$ with $\mathbb{P}^{*}\circ({\cal X}^{*})^{-1}=Q^{*}$ such that, $\mathbb{P}^{*}$-a.s., as $k\to\infty$,

[TABLE]

Let ${\cal D}:=C_{0}^{\infty}(\mathbb{R}^{m})$ be the space of test functions with its dual space given by ${\cal D}^{\prime}$. Denote by $(\cdot,\cdot)$ the dual pair between ${\cal D}$ and ${\cal D}^{\prime}$. We next prove that $\theta^{*}_{k}$ is ${\cal H}_{m}^{1}$-valued. For any $\phi\in{\cal D}$, we define the linear functional ${\cal T}_{k}(\phi):=(\theta_{k}^{*},\phi^{\prime})$ on ${\cal D}$, for $k\geq 1$. Let $(\phi_{l})_{l=1}^{\infty}\subset{\cal D}$ be dense in ${\cal L}_{m}^{2}$ and consider

[TABLE]

Let $\mathbb{E}^{*}$ be the expectation operator under $\mathbb{P}^{*}$. Then, for all $k\geq 1$, it holds that

[TABLE]

and hence $\|{\cal T}_{k}\|<+\infty$, $\mathbb{P}^{*}$-a.s.. Then, by Hahn-Banach theorem, it holds $\mathbb{P}^{*}$-a.s. that ${\cal T}_{k}$ can be extended to be a bounded linear functional on ${\cal L}_{m}^{2}$. Thus, Riesz representation theorem yields the existence of an ${\cal L}_{m}^{2}$-valued random variable $\hat{\theta}_{k}^{*}$ such that ${\cal T}_{k}(\phi)=(\hat{\theta}_{k}^{*},\phi)$ for all $\phi\in{\cal L}_{m}^{2}$, $\mathbb{P}^{*}$-a.s. In particular, ${\cal T}_{k}(\phi)=(\theta_{k}^{*},\phi^{\prime})=(\hat{\theta}_{k}^{*},\phi)$ for any $\phi\in{\cal D}$. Hence, $\theta_{k}^{*}\in{\cal H}_{m}^{1}$, $\mathbb{P}^{*}$-a.s. It follows that $\mathbb{P}^{*}\circ({\cal X}_{k}^{*})^{-1}\in{\cal P}(\Omega_{\infty})$. Moreover, we obtain from (A.4) that

[TABLE]

This implies that $({\theta_{k}^{*}}^{\prime})_{k=1}^{\infty}$ is bounded in $L^{2}((0,T)\times\Omega^{*};dt\otimes d\mathbb{P}^{*})$, and hence ${\theta_{k}^{*}}^{\prime}$ (up to a subsequence) converges weakly to some element $\hat{\theta}^{*}\in L^{2}((0,T)\times\Omega^{*};dt\otimes d\mathbb{P}^{*})$ as $k\to\infty$. Let $\phi\in{\cal D}$. By (A.11), we have that, for all $H\in L^{\infty}(\Omega^{*};\mathbb{P}^{*})$,

[TABLE]

Then $(\theta^{*},\phi^{\prime})=-(\hat{\theta}^{*},\phi)$, $\mathbb{P}^{*}$-a.s. Therefore, using the separability of ${\cal D}$, it holds $\mathbb{P}^{*}$-a.s. that $(\theta^{*},\phi^{\prime})=-(\hat{\theta}^{*},\phi)$ for all $\phi\in{\cal D}$. This gives that ${\theta^{*}}^{\prime}=\hat{\theta}^{*}$, $\mathbb{P}^{*}$-a.s. Thus, $\mathbb{P}^{*}\circ({\cal X}^{*})^{-1}\in{\cal P}(\Omega_{\infty})$.

For $k\geq 1$ and $i\in\mathbb{N}$, let $X_{k}^{*,i}$ be the strong solution of SDE (1) driven by $(\zeta_{k}^{*},W_{k}^{*},\theta_{k}^{*})$. In other words, under $(\Omega^{*},\mathcal{F}^{*},\mathbb{P}^{*})$, $(X_{k}^{*,i}(0),Y_{k}^{*,i}(0))=\zeta_{k}^{*,i}$, and for $t\in(0,T]$,

[TABLE]

Moreover, for $i\in\mathbb{N}$, let $X^{*,i}$ be the strong solution of SDE (1) driven by $(\zeta^{*},W^{*},\theta^{*})$. Under $(\Omega^{*},\mathcal{F}^{*},\mathbb{P}^{*})$, $(X^{*,i}(0),Y^{*,i}(0))=\zeta^{*,i}$, and for $t\in(0,T]$,

[TABLE]

By the assumption (Aε,f,ρ), we obtain that, for all $t\in[0,T]$, $\mathbb{P}^{*}$-a.s.

[TABLE]

Using the Lipschitz property of $\rho:\mathbb{R}^{d}\to\mathbb{R}^{q}$, we obtain from Jensen’s inequality that

[TABLE]

Define $|\tilde{x}|_{N}^{2}:=\frac{1}{N}\sum_{i=1}^{N}|x^{i}|^{2}$ for $\tilde{x}=(x^{1},\ldots,x^{N})\in(\mathbb{R}^{d})^{N}$. It holds that, $\mathbb{P}^{*}$-a.s.

[TABLE]

Therefore, using the convergence results from (A.11), we conclude that, as $k\to\infty$,

[TABLE]

From (A.11) and (A.20), it follows that, $\mathbb{P}^{*}$-a.s., as $k\to\infty$,

[TABLE]

It follows from properties of convex functionals and weak convergence (see, e.g. Theorem 1.4 in De Figueiredo (1991)) that

[TABLE]

Recall the sampled objective functional given in (8), and the square form of the loss function and regularizer given in (5). Note that $Q_{k}=\mathbb{P}^{*}\circ({\cal X}_{k}^{*})^{-1}$, and $\mathbb{P}^{*}\circ({\cal X}^{*})^{-1}=Q^{*}$. Then, it follows from (A.22) that $Q^{*}\in{\cal Q}(\nu)$. Moreover, by Fatou’s lemma, it follows that

[TABLE]

We then deduce that $J_{N}(Q^{*})\leq\alpha_{N}$ by using (A.1). Recall that $Q^{*}\in{\cal Q}(\nu)$ and hence $\alpha_{N}\leq J_{N}(Q^{*})$. Therefore $J_{N}(Q^{*})=\alpha_{N}$, i.e., $Q^{*}\in{\cal Q}(\nu)$ is the optimal relaxed solution of (9). This ends the proof. ∎

The proofs of Lemma 4.2 and Lemma 4.3, rely on the following technical lemma. Its proof is omitted because it is standard, and based on an application of the Itô’s formula and use of the Doob’s maximal inequality (the proof can be provided upon request).

Lemma A.1**.**

Let $\tilde{X}_{N}(t)=(X_{N}^{1}(t),\ldots,X_{N}^{N}(t))$ for $t\in[0,T]$ with $X_{N}^{i}(t)$ satisfying SDE (13). Let (Aε,f,ρ)* and (AΘ)** hold. Then, for all $p\geq 1$, there exists a positive constant $C_{p}$ which is independent of $N$ such that*

[TABLE]

Proof of Lemma 4.2.

It follows from (15) and Chebyshev’s inequality that

[TABLE]

By the assumption (Aε,f,ρ)-(i) and noting that $\zeta^{i}\in\Xi_{K}$ for all $i\geq 1$, the limit (20) follows from Lemma A.1. Using the representation of the empirical measure given by (14), it holds that, for $s,t\in[0,T]$,

[TABLE]

Then, it follows from Chebyshev’s inequality that

[TABLE]

Using the assumption (Aε,f,ρ) and Lemma A.1 with the assumption (AΘ), we deduce the existence of a positive constant $C$ which is independent of $N$ such that

[TABLE]

where, for $\delta>0$, we have defined

[TABLE]

Note that $(W_{N}^{1},\ldots,W_{N}^{N})$ are independent Wiener processes under $Q_{N}$. Note that, it holds that

[TABLE]

Since $W_{1}^{1}$ is a Wiener process under $Q_{1}$, we have $\lim_{\delta\to 0}\sup_{|t-s|\leq\delta}|W_{1}^{1}(t)-W_{1}^{1}(s)|^{2}=0$, $Q_{1}$-a.s.. On the other hand, the BDG inequality yields $\mathbb{E}^{Q_{1}}[\sup_{t\in[0,T]}|W_{1}^{1}(t)|^{2}]\leq C_{T}$ for some $C_{T}>0$ depending on $T$ only. Then, it follows from DCT that $\Upsilon(\delta)\to 0$ as $\delta\to 0$. Hence, the limit (21) follows from (A.27). ∎

Proof of Lemma 4.8.

Define the mappings $\hat{I}_{N},\hat{I}_{*}:\Xi_{K}^{\mathbb{N}}\rightarrow\Omega^{0}_{\infty}\times{\cal P}_{2}(E)$ as follows: for any $\hat{\zeta}\in\Xi_{K}^{\mathbb{N}}$,

[TABLE]

It follows from (52) in (Aν) that $\nu\circ\hat{I}_{N}^{-1}\Rightarrow\nu\circ\hat{I}_{*}^{-1}$ as $N\to\infty$. Observe that $Q_{N}\circ\hat{I}_{N}(\zeta_{N})^{-1}=\{Q_{N}\circ\zeta^{-1}_{N}\}\circ\hat{I}_{N}^{-1}=\nu\circ\hat{I}_{N}^{-1}$, and $Q\circ\hat{I}_{*}(\zeta)^{-1}=\nu\circ\hat{I}_{*}^{-1}$. Then

[TABLE]

Using the inequality ${\cal W}_{\Omega_{\infty},2}(Q_{N}\circ(\zeta_{N},\theta_{N})^{-1},Q\circ(\zeta,\theta)^{-1})\leq{\cal W}_{\Omega_{\infty},2}(Q_{N},Q)$ and the assumption that $\lim_{N\to\infty}{\cal W}_{\Omega_{\infty},2}(Q_{N},Q)=0$, we arrive at

[TABLE]

Combining (A.30) and (A.31), we obtain that $(Q_{N}\circ(\zeta_{N},I_{N}(\zeta_{N}),\theta_{N})^{-1})_{N=1}^{\infty}$ is tight.

We next prove that any convergent subsequence of $(Q_{N}\circ(\zeta_{N},I_{N}(\zeta_{N}),\theta_{N})^{-1})_{N=1}^{\infty}$ has the same weak limit. To start with, let $(N_{k}^{i})_{k=1}^{\infty}$, $i=1,2$, be two subsequences of $\mathbb{N}$ such that $Q_{N_{k}^{i}}\circ(\zeta_{N_{k}^{i}},I_{N_{k}^{i}}(\zeta_{N_{k}^{i}}),\theta_{N_{k}^{i}})^{-1}\Rightarrow\mathbb{P}^{i}\circ(\zeta_{i},J_{i},\theta_{i})^{-1}$ as $k\to\infty$. Here, $(\zeta_{i},J_{i},\theta_{i})$ is a $\Omega^{0}_{\infty}\times{\cal P}_{2}(E)\times{\cal H}^{1}_{m}$-valued random variable defined on some probability space $(\Omega^{i},\mathcal{F}^{i},\mathbb{P}^{i})$. By (A.30), we have that $\mathbb{P}^{i}\circ(\zeta_{i},J_{i})^{-1}=\nu\circ\hat{I}_{*}^{-1}$ for $i=1,2$. It then follows from (A.29) that, for $i=1,2$,

[TABLE]

By applying the Gluing lemma (see Lemma 7.6 in Villani (2003)), there exists a coupling $(J_{1}^{*},J_{2}^{*},\zeta^{*},\theta^{*})$ under some probability space $(\Omega^{*},\mathcal{F}^{*},\mathbb{P}^{*})$ such that $(\zeta^{*},J_{i}^{*},\theta^{*})=(\zeta_{i},J_{i},\theta_{i})$ in law for $i=1,2$. It then follows from (A.32) that $J_{1}^{*}=J_{2}^{*}=I_{*}(\zeta^{*})$, $\mathbb{P}^{*}$-a.s. This yields that $\mathbb{P}^{1}\circ(\zeta_{1},J_{1},\theta_{1})^{-1}=\mathbb{P}^{2}\circ(\zeta_{2},J_{2},\theta_{2})^{-1}$, and hence every convergent subsequence of $(Q_{N}\circ(\zeta_{N},I_{N}(\zeta_{N}),\theta_{N})^{-1})_{N=1}^{\infty}$ admits the same weak limit. Moreover, for $Q\in{\cal Q}(\nu)$, the assumption (Aν) together with Definition 3.1-(i) yields

[TABLE]

and hence $(\zeta,I_{N}(\zeta),\theta)\to(\zeta,I_{*}(\zeta),\theta)$, $Q$-a.s. This concludes the proof that $Q\circ(\zeta,I_{N}(\zeta),\theta)^{-1}\Rightarrow Q\circ(\zeta,I_{*}(\zeta),\theta)^{-1}$ as $N\to\infty$. Moreover, define a specific sequence $(\hat{Q}_{N})_{N=1}^{\infty}\subset{\cal Q}(\nu)$ as $\hat{Q}_{2l-1}:=Q_{2l-1}$ and $\hat{Q}_{2l}=Q$ for all $l\in\mathbb{N}$. We then have that $\hat{Q}_{N}\circ(\zeta_{N},I_{N}(\zeta_{N}),\theta_{N})^{-1}\Rightarrow{Q}\circ(\zeta,I_{*}(\zeta),\theta)^{-1}$. This proves the weak convergence result in the lemma. ∎

Proof of Lemma 4.10.

The uniqueness of a solution to the FPK equation (59) in the trajectory sense follows from Proposition 4.4. We next show the existence. For given $(I_{*},\theta)\in{\cal P}_{2}(E)\times{\cal C}_{m}$, consider the weak solution of the following parameterized SDE defined on a filtered probability space $(\hat{\Omega},\hat{\mathcal{F}},\hat{\mathbb{F}}=(\hat{\mathcal{F}}_{t})_{t\in[0,T]},\hat{\mathbb{P}})$ which supports a $p$-dimensional Brownian motion $\hat{W}=(\hat{W}(t))_{t\in[0,T]}$, and r.v.s $(Y(0),X(0))\in\hat{\mathcal{F}}_{0}$:

[TABLE]

and $(\varepsilon,Y(0),X(0))$ admits the law $I_{*}$. It can then be verified that $\mu(t):=\hat{\mathbb{P}}\circ((\varepsilon,Y(0)),X^{\xi,I_{*},\theta}(t))^{-1}$, for $t\in[0,T]$, satisfies Eq. (59).

Moreover, by Lemma 4.8, $Q_{N}\circ(\mu^{N}(0),\theta_{N})^{-1}\Rightarrow Q\circ(I_{*},\theta)^{-1}$ as $N\to\infty$. It follows from Theorem 4.1 that $\mathbb{Q}^{N}:=Q_{N}\circ(\mu^{N}(0),\theta_{N},\mu^{N})^{-1}\to\hat{\mathbb{P}}\circ(\hat{\mu}_{0},\hat{\theta},\hat{\mu})^{-1}$ in ${\cal P}_{2}({\cal P}_{2}(E)\times{\cal C}_{m}\times\hat{S})$ for some probability space $(\hat{\Omega},\hat{\mathbb{F}},\hat{\mathbb{P}})$, where $\hat{\mathbb{P}}$-a.s., $\hat{\mu}$ is the unique solution of FPK equation (18) with initial condition $\hat{\mu}(0)=\hat{\mu}_{0}$. Note that $(\hat{\mu}_{0},\hat{\theta})$ has the law given by $Q\circ(I_{*},\theta)^{-1}$. Then, from Gluing lemma, there exists a coupling $(\bar{\mu}_{0},\bar{\theta},\bar{\mu}^{1},\bar{\mu}^{2})$ on some probability space $(\bar{\Omega},\bar{\mathcal{F}},\bar{\mathbb{P}})$ such that $\bar{\mathbb{P}}\circ(\bar{\mu}_{0},\bar{\theta},\bar{\mu}^{1})^{-1}=\hat{\mathbb{P}}\circ(\hat{\mu}_{0},\hat{\theta},\hat{\mu})^{-1}$ and $\bar{\mathbb{P}}\circ(\bar{\mu}_{0},\bar{\theta},\bar{\mu}^{2})^{-1}=Q\circ(I_{*},\theta,\mu_{*})^{-1}$. Recall here that, $Q$-a.s., $\mu_{*}$ solves FPK equation (18) with initial condition $\mu_{*}(0)=I_{*}$. Using a similar proof to that of Lemma 4.5, it follows that $(\hat{\mu}_{0},\hat{\theta},\hat{\mu})$ and $(I_{*},\theta,\mu_{*})$ are identical in law. Taking $Q_{N}=Q$ for all $N\geq 1$, we deduce that $\hat{\mathbb{Q}}^{*}=Q\circ(I_{*},\theta,\mu_{*})^{-1}$. ∎

Proofs for the case $\lambda_{2}=0$ in (6). We sketch the proofs for the case $\lambda_{2}=0$ in (6). We impose the assumption that the forcing function $f$ in Section 7 is of the form specified by (89), both in the case of finite and infinite sample size. If the sample size is finite, we can use the HJB equation to establish the existence of minimizers in the feedback form. Then, using the standard verification argument, we can show that the value function of relaxed controls coincides with the one of strong controls. Moreover, given the existence of optimal (feedback) controls, using stochastic maximum principle, we can study the convergence of minimizers as $N$ tends to infinity. For each fixed sample size $N$, we apply the stochastic maximum principle to characterize the optimal control $\theta_{N}=(\theta_{N}^{(1)}(t),\theta_{N}^{(2)}(t))_{t\in[0,T]}$. To adopt this framework, we take the canonical space in (10) as $\Omega_{\infty}=\Xi_{K}^{\mathbb{N}}\times{\cal C}_{p}^{\mathbb{N}}\times L^{1}([0,T];\mathbb{R}^{d}\times\mathbb{R}^{l})$ and use Definition 3.1 on relaxed control without the condition (iii). For $i=1,\ldots,N$, consider the adjoint process $(p_{N}^{i},q_{N}^{i})=(p_{N}^{i}(t),q_{N}^{i}(t))_{t\in[0,T]}\in\mathbb{R}^{d}\times\mathbb{R}^{d\times Nd}$ which satisfies the following linear BSDE:

[TABLE]

where, for $(t,i)\in[0,T]\times\{1,\ldots,N\}$, $W(t)^{\top}=(W^{1}_{N}(t)^{\top},W^{2}(t)^{\top},\ldots,W^{N}(t)^{\top})$, $\partial_{i}R_{N}(t):=\frac{\beta}{N}(X^{i}(t)-Y^{i}(0))$, $S(i,t):=S(t,X^{i}(t),\sum_{j=1}^{N}\rho(X^{j}(t)))$ and $\nabla_{\ell}S(i,t):=\nabla_{\ell}S(t,X^{i}(t),\sum_{j=1}^{N}\rho(X^{j}(t)))$ for $\ell\in\{x,\eta\}$. For $t\in[0,T]$, denote by

[TABLE]

Note that $p_{N}(t),X(t),Y(0)\in\mathbb{R}^{Nd\times 1}$. Let $A^{N}(t)=(A^{N}_{ij}(t))_{N\times N}$ whose $(i,j)$-th entry as $d\times d$-submatrix given by

[TABLE]

Hence, for $A^{N}(t)=(a_{kl}^{N}(t))_{Nd\times Nd}$, we have from the compactness of $\Theta$ and Assumption (Aε,f,ρ) that $|a_{kl}^{N}(t)|\leq C$ for $k=l$; while $|a_{kl}^{N}(t)|\leq C/(Nd)$ for $k\neq l$, where $C>0$ is a constant which depends on $T$ only. Moreover, it holds that

[TABLE]

where $\mathbb{E}_{t}[\cdot]:=\mathbb{E}[\cdot|\mathcal{F}_{t}]$ for $t\in[0,T]$. Here, $\Phi^{N}_{t,T}(A^{N})\in\mathbb{R}^{Nd\times Nd}$ satisfies that, a.s.

[TABLE]

By the uniform estimate on $a_{kl}^{N}=(a_{kl}^{N}(t))_{t\in[0,T]}$ above, for $t\geq 0$ fixed, we also have that

[TABLE]

for all $s\in[0,T-t]$ and $u\in[t+s,T]$. Here, $C>0$ is a constant depending on $T$ only. The optimality of the control $\theta_{N}=(\theta_{N}(t))_{t\in[0,T]}=(\theta_{N}^{(1)}(t),\theta_{N}^{(2)}(t))_{t\in[0,T]}$ yields that

[TABLE]

where $P_{\Theta}$ is the projection on $\Theta$, and

[TABLE]

Note that the functional on RHS of (A.38) is quadratic in the control variable, and hence admits a unique minimizer:

[TABLE]

Let $t\geq 0$ and $\delta>0$. It follows from (A.37) that $\mathbb{E}[\sup_{0\leq s\leq\delta}|\theta_{N}(t+s)-\theta_{N}(t)|]\leq\mathbb{E}[\sup_{0\leq s\leq\delta}|\tilde{\theta}_{N}(t+s)-\tilde{\theta}_{N}(t)|]$. On the other hand, we deduce from (A.39) that, for all $s\in[0,\delta]$,

[TABLE]

and $|\tilde{\theta}^{(2)}_{N}(t+s)-\tilde{\theta}^{(2)}_{N}(t)|\leq\frac{1}{2}\sum_{i=1}^{N}|p_{N}^{i}(t+s)-p_{N}^{i}(t)|$. Below, let $C>0$ be a generic constant which depends on $T,d$ only, but it may be different from line to line. Using the condition satisfied by $S(t,x,\eta)$ in (89), the probabilistic representation (A.34) of $p_{N}(t)$ and the estimate (A.36), we obtain that, for all $s\in[0,\delta]$,

[TABLE]

and meanwhile, it holds that

[TABLE]

Thus, in light of (A.40) and (A.41), we arrive at

[TABLE]

Hence, from (A.42) and the moment estimate of $X^{i}(t)$ for $i=1,\ldots,N$, it deduces that

[TABLE]

This yields from Chebyshev’s inequality that, for any $\varepsilon>0$,

[TABLE]

Note that $\tilde{\theta}_{N}$ is $\Theta$-valued, and $\Theta$ is compact. Then, Lemma A.2 in Barbu et al. (2018) yields the tightness of $(Q_{N})_{N\geq 1}$ with $Q_{N}$ being the relaxed control associated with $\tilde{\theta}_{N}=(\tilde{\theta}_{N}(t))_{t\in[0,T]}$. Therefore, we have shown the compactness of sequence of minimizers for $J_{N}(Q)$ over all $Q\in{\cal Q}(\nu)$. Note also that $J_{N}\overset{\Gamma}{\to}J$, as $N\to\infty$ (c.f. Remark 5.4), and thus the convergent subsequence of minimizers $(Q_{N})_{N\geq 1}$ for the objective functional $J_{N}$ converges to the minimizer $Q$ of the limit objective functional $J$, as $N\to\infty$.