Random Neural Networks for Rough Volatility

TL;DR

This paper introduces a deep learning algorithm to solve complex financial equations related to rough volatility using a special type of neural network.

Contribution

A novel deep learning algorithm using reservoir neural networks is proposed for solving path-dependent PDEs in rough volatility.

Findings

The reservoir neural network approach simplifies the optimization problem to a least-square regression.

Theoretical convergence properties of the proposed method are proven.

Abstract

We construct a deep learning-based numerical algorithm to solve path-dependent partial differential equations arising in the context of rough volatility. Our approach is based on interpreting the PDE as a solution to an BSDE, building upon recent insights by Bayer, Qiu and Yao, and on constructing a neural network of reservoir type as originally developed by Gonon, Grigoryeva, Ortega. The reservoir approach allows us to formulate the optimisation problem as a simple least-square regression for which we prove theoretical convergence properties.

Figures

Empirical convergence of the MSE from Corollary 5.5 under Black-Scholes in terms of the number of hidden nodes (for a fixed grid and number of MC samples). Error bars mark 0.1 and 0.9 quantiles of 20 separate runs of the algorithm. The slope coefficient of the dashed line is obtained through regression of the means of individual runs, while the solid line represents 1/*K* convergence and is shown as a reference

Computational time vs number of dimensions, as in Table [2](#Tab2)

Empirical convergence of MSE under rBergomi in terms of the number of hidden nodes. Error bars mark 0.1 and 0.9 quantiles of 20 separate runs of the algorithm. The slope coefficient of the dashed line is obtained through regression of the means of individual runs, while the solid line represents 1/*K* convergence and is shown as a reference

Full text

Introduction

In recent years, a fundamental shift from classical modelling towards so-called rough stochastic volatility models has happened. These “rough" models were first proposed by Gatheral, Jusselin, Rosenbaum [27] and by Bayer, Gatheral, Friz [4], and have since sparked a great deal of research, because of their ability to capture stylised facts of volatility time series and of option prices more accurately, while remaining parsimonious. In essence, they are a class of continuous-path stochastic volatility models, where the instantaneous volatility is driven by a stochastic process with paths rougher than those of Brownian Motion, typically modelled by a fractional Brownian motion [50] with Hurst parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H\in (0,1)$$\end{document} . The reason for this drastic paradigm shift can be found not only under the historical measure, where the roughness of the time series of daily log-realised variance estimates suggests Hölder regularity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H\approx 0.1$$\end{document} , but also under the pricing measure, where rough volatility models are able to reproduce the power-law behaviour of the ATM volatility skew. Since then, a slew of papers have appeared, providing closed-form expressions for the characteristic functions of rough Heston models [21], machine learning techniques for calibration [39], microstructural foundations [20], option pricing partial differential equations (PDEs) solvers [6, 45], among others. A full overview can be found in the recent monograph [5].

Dating back to Black-Scholes [12], PDEs have been used to model the evolution of the prices of European-style options. However, rough volatility models give rise to a non-Markovian framework, where the value function for a European option is not deterministic anymore, but is instead random and satisfies a backward stochastic partial differential equation (BSPDE) as was shown in [6]. Moreover, even in classical diffusive models, the so-called curse of dimensionality poses a challenge when solving PDEs in high dimension; until recently, only the backward stochastic differential equation (BSDE) approach by [54] was available to tackle this, which is not really feasible in dimension beyond six.

On a positive note, machine learning methods have spread inside quantitative finance in recent years, and neural networks, in particular, have become a powerful tool to overcome problems in high-dimensional situations, because of their superior computational performance across a wide range of applications [15, 30, 59]; more precisely in the context of PDEs, examples of applications thereof can be found in [7, 18, 35, 45, 60, 62]. For a more thorough literature review on the use of neural networks in finance and finance-related PDEs, we refer the reader to the surveys in [8, 29].

In this paper, we focus on the works by Huré, Pham and Warin [41], and by Bayer, Qiu and Yao [6], where the classical backward resolution technique is combined with neural networks to estimate both the value function and its gradient. Not only does this approach successfully reduce the curse of dimensionality, but also appears more effective in both accuracy and computational efficiency than existing Euler-based approaches.

Besides research on numerical aspects, a lot of progress has been made on the theoretical foundations for neural network-based methods, in particular showing that they are able to approximate solutions of certain types of PDEs [19, 33, 43, 57]. These results are significant as they show that deep neural networks can be used to solve complex problems that were previously thought intractable. However, in practice, optimal parameters of any given neural network minimising a loss function ultimately have to be calculated approximately. This is usually done through some kind of stochastic gradient descent (SGD) algorithm, which inadvertently introduces an optimisation error. Because of the non-convexity of the network’s loss surface and the stochastic nature of the SGD, the optimisation error is notoriously hard to treat rigorously. One such attempt by Gonon [31] instead involves the use of neural networks in which only certain weights are trainable and the remaining are randomly fixed. This suggests that these random-weight neural networks are, in fact, capable of learning non-degenerate Black-Scholes-type PDEs without succumbing to the curse of dimensionality. Following this, we combine the classical BSDE approach [14, 54] with random-weight neural networks (RWNNs) [40, 55, 56].

Our final algorithm then reduces to a least-square Monte-Carlo, introduced by Longstaff and Schwartz [48] (see also [1] for related applications), where the usually arbitrary choice of basis is ‘outsourced’ to the reservoir of the corresponding RWNN. The basis is computationally efficient and ultimately allows us to express the approximation error in terms of the number of stochastic nodes in the network. Moreover, vectorisation of the Randomised Least Square along the sampling direction allows us to evaluate the sum of outer tensor products using the einsum function (available in NumPy and PyTorch) and achieve an even greater speed-up. One word of caution though: in our numerical examples, for the rough Bergomi model and for Basket options, computation time is still longer than using Monte Carlo methods, mostly because it does require simulating sample paths. It however opens the gates to (more advanced) numerical schemes for path-dependent partial differential equations, which we plan to investigate more in later projects.

To summarise, in contrast with Bayer-Qiu-Yao [6], our numerical scheme employs RWNNs as opposed to the conventional feed-forward neural networks, resulting in significantly faster training times without sacrificing the accuracy of the scheme. Moreover, this structure allows us to provide error bounds in terms of the number of hidden nodes, granting additional insights into the network’s performance. Given the comparable performance of RWNNs and conventional feed-forward neural networks, we argue that this paper illuminates an essential lesson, namely that the additional complexity of deep neural networks can sometimes be redundant at the cost of precise error bounds. We note in passing that RWNNs have already been used in Finance to price American options [36], for financial data forecasting [47], for PIDEs [33], and we refer the interested reader to [16] for a general overview of their applications in data science.

Moreover, in parallel to our work, Shang, Wang, and Sun [61] combined randomised neural networks with Petrov-Galerkin methods to solve linear and non-linear PDEs. Their method, similar to ours, uses randomly initialised neural networks with trainable linear readouts. Neufeld, Schmocker, and Wu [53] conducted a comprehensive error analysis of the random deep splitting method for non-linear parabolic PDEs and PIDEs, demonstrating high-dimensional problem-solving capabilities.

The paper is structured as follows: Section 2 provides a brief overview of Random-weight Neural Networks (RWNNs), including their key features and characteristics. In Section 3, we outline the scheme for the Markovian case and discuss the non-Markovian case in Section 4. The convergence analysis is presented in Section 5. Additionally, Section 6 presents numerical results, which highlight the practical relevance of the scheme and its performance for different models. Some of the technical proofs are postponed to Appendix B to ease the flow of the paper.

Notations: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^+ = [0,\infty )$$\end{document} ; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\aleph $$\end{document} refers to a random neural network, defined in Section 2; for an open subset \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E\subset \mathbb {R}^d$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1\le p \le \infty $$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s \in \mathbb {N}$$\end{document} we define the Sobolev space

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathcal {W}^{s, p}(E, \mathbb {R}^m) :=\Big \{f \in L^p(E, \mathbb {R}^m): \; \partial _{\textbf{x}}^{\boldsymbol{\alpha }} f \in L^p(E, \mathbb {R}^m), {\text {for all }} |\boldsymbol{\alpha }| \le s\Big \}, $$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\alpha } = \left( \alpha _1, \ldots , \alpha _d\right) $$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\boldsymbol{\alpha }|=\alpha _1+\ldots +\alpha _d$$\end{document} , and the derivatives \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\partial _{\textbf{x}}^{\boldsymbol{\alpha }} f = \partial _{x_1}^{\alpha _1} \dots \partial _{x_d}^{\alpha _d} f$$\end{document} are taken in a weak sense. To be consistent with probabilistic notations—although Machine Learning literature tends to differ—we shall write \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {E}^\Phi [\cdot ]:= \mathbb {E}[\cdot \vert \Phi ]$$\end{document} as the conditional expectation with respect to the random variable  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi $$\end{document} .

Random-Weight Neural Network (RWNN)

Neural networks with random weights appeared in the seminal works by Barron [2, 3], but a modern version was proposed by Huang [40] under the name Extreme learning machine, and today are known under different names: reservoir networks, random feature or random-weight networks; we adopt the latter as it sounds more explicit to us.

Definition 2.1

(Neural network) Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L, N_{0}, \ldots , N_{L} \in \mathbb {N}, \varrho : \mathbb {R}\rightarrow \mathbb {R}$$\end{document} and for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l=$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1, \ldots , L$$\end{document} let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{l}: \mathbb {R}^{N_{l-1}} \rightarrow \mathbb {R}^{N_{l}}$$\end{document} an affine function. A function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F: \mathbb {R}^{N_{0}} \rightarrow \mathbb {R}^{N_{L}}$$\end{document} defined as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ F=w_{L} \circ F_{L-1} \circ \cdots \circ F_{1}, \quad \text{ with } F_{l}=\varrho \circ w_{l} \quad \text{ for } l=1, \ldots , L-1, $$\end{document}

is called a neural network, with activation function  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varrho $$\end{document} applied component-wise. L denotes the total number of layers, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{1}, \ldots , N_{L-1}$$\end{document} denote the dimensions of the hidden layers and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{0}$$\end{document} and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{L}$$\end{document} those of the input and output layers respectively. For each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l\in \{1, \dots , L\}$$\end{document} the affine function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{l}:\mathbb {R}^{N_{l-1}}\rightarrow \mathbb {R}^{N_{l}}$$\end{document} is given as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{l}(\textbf{x}) = \textrm{A}^{(l)} \textbf{x}+ \textrm{b}^{(l)}$$\end{document} , for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}\in \mathbb {R}^{N_{l-1}}$$\end{document} , with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{A}^{(l)} \in \mathbb {R}^{N_{l}\times N_{l-1}} $$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{b}^{(l)} \in \mathbb {R}^{N_{l}}$$\end{document} . For any \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{1, \dots N_{l}\}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\in \{1, \dots , N_{l-1}\}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{i j}^{(l)}$$\end{document} is interpreted as the weight of the edge connecting node i of layer  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l-1$$\end{document} to node j of layer l.

A random-weight neural network (RWNN) is a neural network where the hidden layers are randomly sampled from a given distribution and then fixed; consequently, only the last layer is trained: out of all the parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{A}^{(l)}, \textrm{b}^{(l)})_{l=0,\ldots ,L}$$\end{document} of the L-layered neural network, the parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{A}^{(0)}, \textrm{b}^{(0)}, \ldots , A^{(L-1)}, \textrm{b}^{(L-1)})$$\end{document} are randomly sampled and frozen and only \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{A}^{(L)}, \textrm{b}^{(L)})$$\end{document} from the last layer are trained.

The training of such an RWNN can then be simplified into a convex optimisation problem. This makes the training easier to manage and understand both practically and theoretically. However, by only allowing certain parts of the parameters to be trained, the overall capacity and expressivity are possibly reduced. Although it is still unclear if random neural networks still maintain any of the powerful approximation properties of general deep neural networks, these questions have been addressed to some extent in e.g. [32, 52], where learning error bounds for RWNNs have been proved.

Denote now \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\aleph _{\infty }^{\varrho }(d_0,d_1)$$\end{document} the set of random neural networks from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^{d_0}$$\end{document} to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^{d_1}$$\end{document} , with activation function  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varrho $$\end{document} –and we shall drop the explicit reference to input and output dimensions in the notation whenever they are clear from the context. Moreover, for any \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L, K \in \mathbb {N}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\aleph _{L, K}^\varrho $$\end{document} represents a random neural network with a fixed number of hidden layers L and fixed input and output dimension K for each hidden layer. We now give a precise definition of a single layer \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\aleph ^\varrho _{K}:=\aleph ^\varrho _{1, K}$$\end{document} , which we will use for our approximation.

Definition 2.2

(Single layer RWNN) Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\widetilde{\Omega }, \widetilde{\mathcal {F}}, \widetilde{\mathbb {P}})$$\end{document} be a probability space on which the iid random variables on a bounded domain \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{A}_{k}: \widetilde{\Omega } \rightarrow \mathcal {S}\subset \mathbb {R}^{d}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_{k}: \widetilde{\Omega } \rightarrow \mathscr {S} \subset \mathbb {R}$$\end{document} , respectively corresponding to weights and biases, are defined. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\phi }=\left\{ \phi _{k}\right\} _{k \ge 1}$$\end{document} denote a sequence of random basis functions, where each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi _{k}: \mathbb {R}^{d} \rightarrow \mathbb {R}$$\end{document} is of the form

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \phi _{k}(\textbf{x}):=\varrho \left( \textrm{A}_{k}^{\top } \textbf{x}+ b_{k}\right) , \qquad x\in \mathbb {R}^d, $$\end{document}

with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varrho :\mathbb {R}\rightarrow \mathbb {R}$$\end{document} a Lipschitz continuous activation function. For an output dimension m and K hidden units, we define the reservoir or random basis as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi _{K}:=\phi _{1:K}=(\phi _1,\dots ,\phi _K)$$\end{document} and the random network \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\aleph ^\varrho _K$$\end{document} with parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Theta =\left( \theta _{1}, \ldots , \theta _{m}\right) ^\top \in \mathbb {R}^{m\times K}$$\end{document} as the map

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \aleph ^\varrho _K: \textbf{x}\mapsto \Psi _K(\textbf{x};\Theta ):=\Theta \Phi _K(\textbf{x}). $$\end{document}

Thus, for each output dimension \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\in \{1,\dots ,m\}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\aleph ^\varrho _K$$\end{document} produces a linear combination of the first K random basis functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _j^\top \phi _{1:K}:=\sum _{k=1}^{K} \theta _{j,k} \phi _{k}$$\end{document} .

Remark 2.3

In this paper, we will make use of the more compact vector notation

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \Phi _K: \mathbb {R}^d \ni \textbf{x}\mapsto \boldsymbol{\varrho }(\textrm{A}\textbf{x}+\textrm{b}) \in \mathbb {R}^{K}, $$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\varrho }:\mathbb {R}^K\rightarrow \mathbb {R}^K$$\end{document} acts component-wise \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\varrho }(\textbf{y}):=(\varrho (y_1), \dots \varrho (y_K))$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{A}: \widetilde{\Omega } \rightarrow \mathbb {R}^{K\times d}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{b}: \widetilde{\Omega } \rightarrow \mathbb {R}^K$$\end{document} are the random matrix and bias respectively.

Derivatives of ReLu-RWNN

In recent years ReLu neural networks have been predominately used in deep learning, because of their simplicity, efficiency and ability to address the so-called vanishing gradient problem [46]. In many ways, ReLu networks also give a more tractable structure to the optimisation problem compared to their smooth counterparts such as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tanh $$\end{document} and sigmoid. Gonon, Grigoryeva and Ortega [32] derived error bounds to the convergence of a single layer RWNN with ReLu activations. Now, while \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{{\,\mathrm{\varsigma }\,}}(y):=\max \{y,0\}}$$\end{document} is performing well numerically, it is, however, not differentiable at zero (see [11] for a short exposition on the chain-rule in ReLu networks). As ReLu-RWNNs will be used in our approach to approximate solutions of partial differential equations, a discussion on its derivatives is in order. To that end we let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\boldsymbol{\varsigma }}\,}}(\textbf{y}):=({{\,\mathrm{\varsigma }\,}}(y_1),\dots ,{{\,\mathrm{\varsigma }\,}}(y_K))$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{H}(y)=1\hspace{-2.1mm}{1}_{(0,\infty )}(\textbf{y})\in \mathbb {R}^K$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{y}\in \mathbb {R}^K$$\end{document} , where the indicator function is again applied component-wise.

Lemma 2.4

For any linear function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell (\textbf{x})=\textrm{A}\textbf{x}+ \textrm{b}$$\end{document} , with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{A}\in \mathbb {R}^{K\times d}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{b}\in \mathbb {R}^K$$\end{document} , then

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \nabla _x({{\,\mathrm{\boldsymbol{\varsigma }}\,}}\circ \ell )(\textbf{x}) = \operatorname {diag}(\boldsymbol{H}(\textrm{A}\textbf{x}+\textrm{b}))\textrm{A}, \qquad {\text {for a.e. }} \textbf{x}\in \mathbb {R}^d. $$\end{document}

Proof

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {A}:=\left\{ \textbf{x}\in \mathbb {R}^d: ({{\,\mathrm{\boldsymbol{\varsigma }}\,}}\circ \ell )(\textbf{x}){=}0\right\} {=}\left\{ \textbf{x}\in \mathbb {R}^d:\ell (\textbf{x})\le 0\right\} $$\end{document} . Then \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${({{\,\mathrm{\boldsymbol{\varsigma }}\,}}\circ \ell )(\textbf{x})=\ell (\textbf{x})}$$\end{document} for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}\in \mathbb {R}^d\setminus \mathcal {A}$$\end{document} . Since  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell $$\end{document} is Lipschitz, differentiability on level sets [22, Section 3.1.2, Corollary I] implies that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla _{\textbf{x}}\left( {{\,\mathrm{\boldsymbol{\varsigma }}\,}}\circ \ell \right) (\textbf{x}) = \boldsymbol{0}\in \mathbb {R}^d$$\end{document} for almost every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}\in \mathcal {A}$$\end{document} , and hence

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \nabla _{\textbf{x}}({{\,\mathrm{\boldsymbol{\varsigma }}\,}}\circ \ell )(\textbf{x})&=\operatorname {diag}\left( 1\hspace{-2.1mm}{1}_{\{\ell (\textbf{x})\in \mathbb {R}^d\setminus \mathcal {A}\}}\right) \nabla _{\textbf{x}}\ell (\textbf{x}) =\operatorname {diag}\left( 1\hspace{-2.1mm}{1}_{(0,\infty )}(\ell (\textbf{x}))\right) \nabla _{\textbf{x}}\ell (\textbf{x}) \\&=\operatorname {diag}(\boldsymbol{H}(\textrm{A}\textbf{x}+\textrm{b}))\textrm{A}. \end{aligned}$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Thus by Lemma 2.4, the first derivative of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Psi (\cdot ;\Theta )\in \aleph ^{{{\,\mathrm{\varsigma }\,}}}_K$$\end{document} is equal to

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \nabla _{\textbf{x}}\Psi _K(\textbf{x}; \Theta ) = \Theta \operatorname {diag}(\boldsymbol{H}(\textrm{A}\textbf{x}+\textrm{b}))\textrm{A}\qquad {\text {for a.e. }} \textbf{x}\in \mathbb {R}^d. \end{aligned}$$\end{document}

The above statements hold almost everywhere, it is thus appropriate we introduce a notion of approximate differentiability.

Definition 2.5

(Approximate limit, [22, Section 1.7.2]) Consider a Lebesgue-measurable set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E \subset \mathbb {R}^d$$\end{document} , a measurable function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f: E \rightarrow \mathbb {R}^m$$\end{document} and a point \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_0 \in E$$\end{document} . We say \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l \in \mathbb {R}^m$$\end{document} is the approximate limit of f at  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_0$$\end{document} , and write \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ap}\lim _{\textbf{x}\rightarrow x_0} f(\textbf{x})=l$$\end{document} , if for each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varepsilon >0$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \lim _{r \downarrow 0} \frac{\lambda \left( \mathcal {B}_r(\textbf{x}_0) \cap \left\{ \textbf{x}\in E:\;|f(\textbf{x})-l| \ge \varepsilon \right\} \right) }{\lambda (\mathcal {B}_r(\textbf{x}_0))}=0, $$\end{document}

with  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda $$\end{document} the Lebesgue measure and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {B}_r(\textbf{x}_0)$$\end{document} the closed ball with radius \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r>0$$\end{document} and center  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_0$$\end{document} .

Definition 2.6

(Approximate differentiability, [22, Section 6.1.3]) Consider a measurable set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E\subset \mathbb {R}^d$$\end{document} , a measurable map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f:E\rightarrow \mathbb {R}^m$$\end{document} and a point \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_0\in E$$\end{document} . The map f is approximately differentiable at  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_0$$\end{document} if there exists a linear map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{D}_{\textbf{x}}:\mathbb {R}^d\rightarrow \mathbb {R}^m$$\end{document} such that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \textrm{ap}\lim _{\textbf{x}\rightarrow x_{0}} \frac{f(\textbf{x})-f(\textbf{x}_{0})-\textrm{D}_{\textbf{x}}(\textbf{x}-\textbf{x}_{0})}{|\textbf{x}-\textbf{x}_{0}|}=0. $$\end{document}

Then  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{D}_{\textbf{x}}$$\end{document} is called the approximate differential of f at  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_0$$\end{document} . We call f approximately differentiable almost everywhere if its approximately derivative exists almost everywhere.

Remark 2.7

The usual rules from classical derivatives, such as the uniqueness of the differential, and differentiability of sums, products and quotients, apply to approximately differentiable functions. Moreover, the chain rule applies to compositions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varphi \circ f$$\end{document} when f is approximately differentiable at  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_0$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varphi $$\end{document} is classically differentiable at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f(\textbf{x}_0)$$\end{document} .

Remark 2.8

([22, Theorem 4, Section 6.1.3]) For \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f\in \mathcal {W}_{\textrm{loc}}^{1, p}\left( \mathbb {R}^{d}\right) $$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1 \le p \le \infty $$\end{document} , f is approximately differentiable almost everywhere and its approximate derivative equals its weak derivative almost everywhere. We will thus use the operator  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{D}_{\textbf{x}}$$\end{document} to denote the weak derivative and approximate derivative interchangeably, to distinguish them from the classical derivative denoted by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla $$\end{document} .

Lemma 2.9

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E\subset \mathbb {R}^d$$\end{document} be a measurable set with finite measure, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X:\Omega \rightarrow E$$\end{document} a continuous random variable on some probability space \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\Omega , \mathcal {F}, \mathbb {P})$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varphi \in \mathcal {C}^1(\mathbb {R}^m)$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi _{\textrm{ap}}: E\rightarrow \mathbb {R}^m$$\end{document} an approximately differentiable function, and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi $$\end{document} its \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {C}^1(\mathbb {R}^d; \mathbb {R}^m)$$\end{document} extension to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^d$$\end{document} . Then \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {E}[\varphi (\textrm{D}_{x}\Phi _{\textrm{ap}}(X))] = \mathbb {E}[\varphi (\nabla _x\Phi (X))] $$\end{document} .

Proof

By [23, Theorem 3.1.6] a function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi _{\textrm{ap}}: E \rightarrow \mathbb {R}^{m}$$\end{document} is approximately differentiable almost everywhere if for every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varepsilon >0$$\end{document} there is a compact set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F \subset E$$\end{document} such that the Lebesgue measure \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda (E \backslash F)<\varepsilon $$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left. \Phi _{\textrm{ap}}\right| _{F}$$\end{document} is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {C}^{1}$$\end{document} and there exists a  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {C}^1$$\end{document} -extension on  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^d$$\end{document} . Since  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varphi $$\end{document} is everywhere differentiable, it maps null-sets to null-sets [58, Lemma 7.25]. The claim follows since  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {P}$$\end{document} is absolutely continuous with respect to the Lebesgue measure  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda $$\end{document} , X being a continuous random variable. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Corollary 2.10

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E\subset \mathbb {R}^d$$\end{document} be a measurable set with finite measure, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X:\Omega \rightarrow E$$\end{document} a continuous random variable on some probability space \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\Omega , \mathcal {F}, \mathbb {P})$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varphi \in \mathcal {C}^1(\mathbb {R}^m)$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi :E\rightarrow \mathbb {R}^m$$\end{document} an approximately differentiable function, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Psi \in \mathcal {W}^{1,p}(E,\mathbb {R}^m)$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\ge 1$$\end{document} such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi = \Psi $$\end{document} almost everywhere. Then \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {E}[\varphi (\textrm{D}_{\textbf{x}}\Phi (X))] = \mathbb {E}[\varphi (\textrm{D}_{\textbf{x}}\Psi (X))]$$\end{document} .

Proof

This is a direct consequence of Lemma 2.9, after noting that the two notions of derivatives are the same on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {W}^{1,p}(E,\mathbb {R}^m)$$\end{document} (see Remark 2.8). \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

From a practical perspective, the second-order derivative of the network with respect to the input will be zero for all intents and purposes. However, as will become apparent in Lemma 5.7, we need to investigate it further, in particular the measure zero set of points where ReLu-RWNN is not differentiable. Rewriting the diagonal operator in terms of the natural basis \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{e_{i}\}$$\end{document} and evaluating the function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{H}$$\end{document} component-wise yields

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \nabla _{\textbf{x}}\Psi _K(\textbf{x}; \Theta )=\Theta \left( \sum _{j=1}^{K} e_{j} e_{j}^\top H\left( e_{j}^\top \textrm{A}\textbf{x}+b_{j}\right) \right) \textrm{A}. $$\end{document}

The i-th component of the second derivative is thus

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left[ \nabla _{\textbf{x}}^2\Psi _K(\textbf{x}; \Theta )\right] _i&= \Theta \left( \sum _{j=1}^{K} e_{j} e_{j}^\top a_{j i}H'\left( e_{j}^\top \textrm{A}\textbf{x}+b_{j}\right) \right) \textrm{A}\\&= \Theta \operatorname {diag}\left( a_{i}\right) \operatorname {diag}\left( \boldsymbol{H}'(\textrm{A}\textbf{x}+\textrm{b})\right) \textrm{A}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a_i$$\end{document} denotes the i-th column of the matrix  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{A}$$\end{document} . Next, we let

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \delta _0^{\varepsilon }(\textbf{x}) :=\frac{H(\textbf{x})-H(\textbf{x}-\varepsilon )}{\varepsilon } \end{aligned}$$\end{document}

for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}\in \mathbb {R}$$\end{document} and define the left derivative of H as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H'=\lim _{\varepsilon \downarrow 0}\delta _0^\varepsilon =\delta _0$$\end{document} in the distributional sense. This finally gives the second derivative of the network:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left[ \nabla _{\textbf{x}}^2\Psi _K(\textbf{x}; \Theta )\right] _i = \Theta \operatorname {diag}\left( a_{i}\right) \operatorname {diag}\left( \boldsymbol{\delta }_0(\textrm{A}\textbf{x}+\textrm{b})\right) \textrm{A}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\delta }_0$$\end{document} denotes the vector function applying \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta _0$$\end{document} component-wise.

Randomised Least Squares (RLS)

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y\in \mathbb {R}^d$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X\in \mathbb {R}^k$$\end{document} random variables on some probability space \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\Omega , \mathcal {F}, \mathbb {P})$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\beta }\in \mathbb {R}^{d\times k}$$\end{document} a deterministic matrix. If the loss function is the mean square error (MSE), the randomised least square estimator reads

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \nabla _{\boldsymbol{\beta }} \mathbb {E}\left[ \Vert Y-\boldsymbol{\beta }X\Vert ^2\right]&= \nabla _{\boldsymbol{\beta }} \mathbb {E}[(Y-\boldsymbol{\beta }X)^\top (Y-\boldsymbol{\beta }X)] \\&= \mathbb {E}\left[ \nabla _{\boldsymbol{\beta }} (Y^\top Y - Y^\top \boldsymbol{\beta }X - X^\top \boldsymbol{\beta }^\top Y + X^\top \boldsymbol{\beta }^\top \boldsymbol{\beta }X)\right] \\&= \mathbb {E}\left[ 2\boldsymbol{\beta }XX^\top - 2YX^\top \right] , \end{aligned}$$\end{document}

which gives the minimiser1 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\beta }= \mathbb {E}\left[ YX^\top \right] \mathbb {E}\left[ XX^\top \right] ^{-1}$$\end{document} , and its estimator

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\widehat{\boldsymbol{\beta }}} :=\left( \sum ^n_{j=1} Y_j X_{j}^{\top }\right) \left( \sum ^n_{j=1} X_jX_j^\top \right) ^{-1}. \end{aligned}$$\end{document}

Depending on the realisation of the reservoir of the RWNN, the covariates of X may be collinear, so that X is close to rank deficient. A standard remedy is to use the Ridge regularised version [37] of the estimator

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ {\widehat{\boldsymbol{\beta }}}_R = \left( \sum ^n_{j=1} Y_j X_{j}^{\top }\right) \left( \sum ^n_{j=1} X_jX_j^\top +\lambda I\right) ^{-1}, \quad {\text {for }} \lambda >0, $$\end{document}

which results in a superior, more robust performance in our experiments.

Remark 2.11

The above derivation holds true for the approximate derivative  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{D}_{x}$$\end{document} as well because all operations above hold for approximately differentiable functions (Remark 2.7).

Remark 2.12

At first glance, the form of the RLS estimator in (2.3) suggests that the sum of outer products over  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n>>1$$\end{document} samples may be computationally expensive. In practice, however, this operation can be implemented efficiently by exploiting the tensor functionalities provided by libraries such as NumPy and PyTorch. In particular, the einsum function enables an efficient evaluation of the required sum of outer products, thereby further optimising the overall computation. Implementation details are provided in the accompanying code, available at ZuricZ/RWNN_PDE_solver.

The Markovian Case

Let the process \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{X}$$\end{document} of the traded and non-traded components of the underlying under the risk-neutral measure \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {Q}$$\end{document} be given by the following d-dimensional SDE:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{X}_s^{t,\textbf{x}} = \textbf{x}+ \int _t^s \mu (r, \textbf{X}_r^{t,\textbf{x}})\textrm{d}r + \int _t^s \Sigma \left( r, \textbf{X}_r^{t,\textbf{x}}\right) \textrm{d}W_r, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu : [0, T]\times \mathbb {R}^d\rightarrow \mathbb {R}^d$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma :[0, T]\times \mathbb {R}^d\rightarrow \mathbb {R}^{d\times d}$$\end{document} adhere to Assumption 3.1, and W is a standard d-dimensional Brownian motion on the probability space \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\Omega , \mathcal {F}, \mathbb {Q})$$\end{document} equipped with the natural filtration \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {F}= \{\mathcal {F}_t\}_{0\le t\le T}$$\end{document} of W. By the Feynman-Kac formula, options whose discounted expected payoff under  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {Q}$$\end{document} can be represented as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} u(t, \textbf{x})=&\mathbb {E}\left[ \int _t^T \textrm{e}^{-r(s-t)} f\left( s, \textbf{X}_s^{t, \textbf{x}}\right) \textrm{d}s + \textrm{e}^{-r(T-t)} g\left( \textbf{X}_T^{t, \textbf{x}}\right) \right] \\&\quad for all (t, \textbf{x}) \in [0, T] \times \mathcal {A}, \end{aligned}$$\end{document}

for  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {A}\subset \mathbb {R}^d$$\end{document} with interest rate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r\ge 0$$\end{document} and continuous functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f:[0,T]\times \mathbb {R}^d\rightarrow \mathbb {R}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g:\mathbb {R}^d\rightarrow \mathbb {R}$$\end{document} can be viewed as solutions to the Cauchy linear parabolic PDE

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{array}{rl} \partial _t u + \mathcal {L}u + f - ru= 0, & \text{ on } [0, T) \times \mathcal {A}, \\ u(T, \cdot )= g, & \text{ on } \mathcal {A}, \end{array} \right. \end{aligned}$$\end{document}

where

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {L}u :=\frac{1}{2}\operatorname {Tr} \left( \Sigma \Sigma ^\top \nabla _{\textbf{x}}^2 u\right) + (\nabla _{\textbf{x}} u)\mu , \qquad {\text {on }}[0,T)\times \mathcal {A}, \end{aligned}$$\end{document}

is the infinitesimal generator associated with diffusion (3.1). In this Markovian setting, we thus adopt a setup similar to [41] and consider a slightly more general PDE

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{array}{r@{\;}ll} \partial _t u(t,\textbf{x}) + \mathcal {L}u(t,\textbf{x}) + f\Big (t, \textbf{x}, u(t,\textbf{x}), \nabla _{\textbf{x}} u(t,\textbf{x}) \cdot \Sigma (t,\textbf{x})\Big ) =& 0, & {\text {on }}[0, T) \times \mathcal {A},\\ u(T, \cdot ) =& g, & {\text {on }}\mathcal {A}, \end{array} \right. \end{aligned}$$\end{document}

with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f:[0,T]\times \mathbb {R}^d \times \mathbb {R}\times \mathbb {R}^{d} \rightarrow \mathbb {R}$$\end{document} such that Assumption 3.1 is satisfied, which guarantees existence and uniqueness of the solution to the corresponding BSDE [54, Section 4].

Assumption 3.1

(Well-posedness of the FBSDE system (3.1)-(3.4)) The drift \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu : [0, T]\times \mathbb {R}^d\rightarrow \mathbb {R}^d$$\end{document} and the diffusion coefficient \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma :[0, T]\times \mathbb {R}^d\rightarrow \mathbb {R}^d\times \mathbb {R}^d$$\end{document} satisfy global Lipschitz conditions. Moreover,

• (i)there exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_f>0$$\end{document} such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \sup _{0 \le t \le T}\Vert f(t, 0,0,0)\Vert <\infty $$\end{document} and, for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(t_1, x_1, y_1, z_1)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(t_2, x_2, y_2, z_2)$$\end{document} in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[0, T] \times \mathbb {R}^d \times \mathbb {R}\times \mathbb {R}^{d}$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\left| f\left( t_2, x_2, y_2, z_2\right) -f\left( t_1, x_1, y_1, z_1\right) \right| \\&\le L_f\left( \sqrt{\left| t_2-t_1\right| }+\left| x_2-x_1\right| +\left| y_2-y_1\right| +\left| z_2-z_1\right| \right) , \end{aligned}$$\end{document}

• (ii)The function g has at most linear growth condition.

The corresponding second-order generator is again given by (3.2). The following assumption is only required to cast the problem into a regression. Otherwise, the optimisation (below) can still be solved using other methods, such as stochastic gradient descent. Another solution would be to use the so-called splitting method to linearise the PDE (as in [7] and the references therein for example).

Assumption 3.2

The function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f:[0,T]\times \mathbb {R}^d \times \mathbb {R}\times \mathbb {R}^{d}\times \mathbb {R}^{d} \rightarrow \mathbb {R}$$\end{document} has an affine structure in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{y}\in \mathbb {R}^m$$\end{document} and in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{z},\textbf{v}\in \mathbb {R}^{d}$$\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ f\left( t, \textbf{x}, \textbf{y}, \textbf{z}, \textbf{v}\right) = a(t,\textbf{x})\textbf{y}+ b(t, \textbf{x})\textbf{z}+ c(t, \textbf{x}) \textbf{v}+ \widetilde{f}(t, \textbf{x}), $$\end{document}

for some real-valued functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a,b,c,\widetilde{f}$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[0,T]\times \mathbb {R}^d$$\end{document} that map to conformable dimensions.

Random Weighted Neural Network Scheme

The first step in so-called deep BSDE schemes [18, 35, 41] is to establish the BSDE associated with the PDE (3.3) and the process (3.1) through the non-linear Feynman-Kac formula. By [54] there exist \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {F}$$\end{document} -adapted processes (Y, Z), which are unique solutions to the BSDE

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_{t}=g\left( \textbf{X}_{T}\right) +\int _{t}^{T} f\left( s, \textbf{X}_{s}, Y_{s}, Z_{s}\right) \textrm{d}s-\int _{t}^{T} Z_{s} \textrm{d}W_{s}, \qquad {\text {for any }}t \in [0,T], \end{aligned}$$\end{document}

and which are connected to the PDE (3.3) via

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ Y_t = u(t, \textbf{X}_t) \qquad {\text {and}} \qquad Z_t = \nabla _x u(t,\textbf{X}_t)\cdot \Sigma (t,\textbf{X}_t). $$\end{document}

with terminal condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$u(T, \cdot )=g$$\end{document} . Next, the BSDE (3.4) is rewritten in forward form

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_{t}=Y_0 - \int _{0}^{t} f\left( s, \textbf{X}_{s}, Y_{s}, Z_{s}\right) \textrm{d}s + \int _{0}^{t} Z_{s} \textrm{d}W_{s}, \qquad {\text {for any }}t \in [0,T], \end{aligned}$$\end{document}

and both processes are discretised according to the Euler-Maruyama scheme. To this end let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi :=\left\{ 0=t_{0}<t_{1}<\ldots <t_{N}=T\right\} $$\end{document} be a partition of the time interval [0, T] with modulus \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\pi |=\max _{i=\{0,1, \dots , N-1\}} \delta _{i}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta _{i}:=t_{i+1}-t_{i}$$\end{document} . Then the scheme is given by

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{array}{r@{\;}l} \textbf{X}_{t_{i+1}} & = \textbf{X}_{t_{i}} + \mu (t_{i}, \textbf{X}_{t_{i}})\delta _{i}+ \Sigma (t_{i}, \textbf{X}_{t_{i}})\Delta ^{W}_{i}, \\ Y_{t_{i+1}} & = Y_{t_{i}} - f\left( t_{i}, \textbf{X}_{t_{i}}, Y_{t_{i}}, Z_{t_i}\right) \delta _{i}+ Z_{t_i}\Delta ^{W}_{i}, \end{array} \right. \end{aligned}$$\end{document}

where naturally \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta ^{W}_{i}:=W_{t_{i+1}} - W_{t_i}$$\end{document} . Then for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{N-1,\dots ,0\}$$\end{document} we approximate  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$u(t_i,\cdot )$$\end{document} with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathfrak {U}_i(\cdot ;\Theta ^i)\in \aleph _{K}^\varrho $$\end{document} and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_{t_i}$$\end{document} as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{array}{r@{\;}l@{\;}l} u(t_i,\textbf{X}_{t_i}) = Y_{t_i} & \approx \mathfrak {U}_i(\textbf{X}_{t_i};\Theta ^i) & :=\Theta ^i \Phi _K^i(\textbf{X}_{t_i}),\\ Z_{t_i} & \approx \mathfrak {Z}_{i}(\textbf{X}_{t_i}) & :=\textrm{D}_{\textbf{x}}\mathfrak {U}_i(\textbf{X}_{t_i};\Theta ^i)\cdot \Sigma (t_i,\textbf{X}_{t_i}) = \Theta ^i \textrm{D}_{\textbf{x}}\Phi _K^i(\textbf{X}_{t_i})\cdot \Sigma (t_i,\textbf{X}_{t_i}). \end{array} \end{aligned}$$\end{document}

Recall that the derivative  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathfrak {Z}_{i}(\textbf{X}_{t_i})$$\end{document} is the approximate derivative from Definition 2.6. The following formulation of the loss function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell $$\end{document} using the approximate derivative is sensible by Lemma 2.9: notice that for the optimal parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Theta ^{i+1, *}$$\end{document} in step \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(i+1)$$\end{document} , the optimal approximation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{\mathfrak {U}}}_{i+1}(\textbf{X}_{t_{i+1}}):=\mathfrak {U}_{i+1}(\textbf{X}_{t_{i+1}}; \Theta ^{i+1, *})$$\end{document} does not depend on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Theta ^i$$\end{document} , hence under Assumption 3.2 with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=0$$\end{document} the loss function at the i-th discretisation step reads

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ell (\Theta ^i)&:=\mathbb {E}^\Phi \left[ \left\| {\widehat{\mathfrak {U}}}_{i+1}(\textbf{X}_{t_{i+1}}) - \left[ \mathfrak {U}_i(\textbf{X}_{t_i};\Theta ^i) - f\Big (t_{i}, \textbf{X}_{t_{i}}, \mathfrak {U}_i(\textbf{X}_{t_i};\Theta ^i), \mathfrak {Z}_{i}(\textbf{X}_{t_i}; \Theta ^i)\Big )\delta _{i}+ \mathfrak {Z}_{i}(\textbf{X}_{t_i}; \Theta ^i)\Delta ^{W}_{i}\right] \right\| ^2\right] \\&= \mathbb {E}^\Phi \left[ \left\| {\widehat{\mathfrak {U}}}_{i+1}(\textbf{X}_{t_{i+1}}) - \left[ (\mathfrak {U}_i(\textbf{X}_{t_i};\Theta ^i) - \left( a_{i}\mathfrak {U}_i(\textbf{X}_{t_i};\Theta ^i) + b_{i}\mathfrak {Z}_{i}(\textbf{X}_{t_i}; \Theta ^i) + \widetilde{f}_{i}\right) \delta _{i}+ \mathfrak {Z}_{i}(\textbf{X}_{t_i}; \Theta ^i)\Delta ^{W}_{i}\right] \right\| ^2\right] \\&= \mathbb {E}^\Phi \left[ \left\| {\widehat{\mathfrak {U}}}_{i+1}(\textbf{X}_{t_{i+1}}) + \widetilde{f}_{i}\delta _{i}- \Theta ^i\Big \{(1-a_{i}\delta _{i})\Phi _K^i(\textbf{X}_{t_i}) + \textrm{D}_{x}\Phi _K^i(\textbf{X}_{t_i}) \Sigma _i \left( b_{i}\delta _{i}+ \Delta ^{W}_{i}\right) \Big \} \right\| ^2\right] \\&= \mathbb {E}^\Phi \left[ \left\| \textrm{Y}^i - \Theta ^i \textrm{X}^i \right\| ^2\right] \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{i} :=p(t_i, \textbf{X}_{t_i})$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p \in \{a, b, \widetilde{f}, \Sigma \}$$\end{document} , and the expectation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {E}^\Phi $$\end{document} is of course conditional on the realisation of the random basis \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi _K^i$$\end{document} , i.e., conditional on the random weights and biases of the RWNN. Furthermore, we used the notations

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textrm{Y}^i :={\widehat{\mathfrak {U}}}_{i+1}(\textbf{X}_{t_{i+1}}) + \widetilde{f}_{i}\delta _{i}\qquad {\text {and}}\qquad \textrm{X}^i :=(1-a_{i}\delta _{i})\Phi _K(\textbf{X}_{t_i}) + \textrm{D}_{\textbf{x}}\Phi _K(\textbf{X}_{t_i})\cdot \Sigma _i \left( b_{i}\delta _{i}+ \Delta ^{W}_{i}\right) . \end{aligned}$$\end{document}

The problem can now be solved via least squares from Section 2.2, yielding the estimator

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \Theta ^{i,*} = \mathbb {E}^\Phi \left[ \textrm{Y}^i\textrm{X}^{i\top }\right] \mathbb {E}^\Phi \left[ \textrm{X}^i\textrm{X}^{i\top }\right] ^{-1}. $$\end{document}

Algorithm

We now summarise the algorithmic procedure of our RWNN scheme. See how the algorithm resembles the Least-Square Monte-Carlo method of [48] after considering sample estimator version of RLS from Section 2.2: Algorithm 1RWNN scheme

Remark 3.3

We discuss the choice of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R>0$$\end{document} from Algorithm 1 in different practical scenarios in Section 6. We find that the scheme remains robust across different choices of support intervals as long as it aligns with the magnitude of the expected output.

The Non-Markovian Case

We now consider a stochastic volatility model under a risk-neutral measure so that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{X}=(X, V)$$\end{document} , where the dynamics of log-price process X are given by

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textrm{d}X_{s}^{t, x} = \left( r-\frac{V_{s}}{2}\right) \textrm{d}s + \sqrt{V_{s}}\Big (\rho _1 \textrm{d}W^1_{s} + \rho _2 \textrm{d}W^2_{s}\Big ), \qquad 0 \le t \le s \le T, \end{aligned}$$\end{document}

starting from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{t}^{t, x} =x\in \mathbb {R}$$\end{document} , with interest rate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r\in \mathbb {R}$$\end{document} , correlation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _1 \in [-1,1]$$\end{document} , and denote \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _2:=\sqrt{1-\rho _1^{2}}$$\end{document} , and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W^1, W^2$$\end{document} are two independent Brownian motions. We allow for a general variance process process V, satisfying the following:

Assumption 4.1

The process V has continuous trajectories, is non-negative almost surely, adapted to the natural filtration of  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W^1$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {E}\left[ \int _{0}^{t} V_{s} \textrm{d}s\right] $$\end{document} is finite for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\ge 0$$\end{document} .

By no-arbitrage, the fair price of a European option with payoff \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h:\mathbb {R}^+\rightarrow \mathbb {R}^+$$\end{document} reads

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ u(t, x) :=\mathbb {E}\left[ \textrm{e}^{-r(T-t)} h\left( \textrm{e}^{X_{T}^{t, x}+r T}\right) \Bigg \vert \mathcal {F}_{t}\right] , \quad {\text {for all }}(t, x) \in [0, T] \times \mathbb {R}, $$\end{document}

subject to (4.1). Since  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{X}$$\end{document} is not Markovian, one cannot characterise the value function u(t, x) via a deterministic PDE. Bayer, Qiu and Yao [6] proved that u can be viewed as a random field which, together with another random field  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi $$\end{document} , satisfies the backward stochastic partial differential equation (BSPDE)

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} -\textrm{d}u(t,x) =&\left[ \frac{V_{t}}{2} \partial _x^{2} u(t, x)+\rho \sqrt{V_{t}} \partial _x \psi (t,x)-\frac{V_{t}}{2} \partial _x u(t,x)-r u(t,x)\right] \textrm{d}t\nonumber \\&-\psi (t,x) \textrm{d}W^1_{t}, \end{aligned}$$\end{document}

in a distributional sense for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(t, x) \in [0, T) \times \mathbb {R}$$\end{document} , with boundary condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${u(T, x)=h\left( \textrm{e}^{x+r T}\right) }$$\end{document} where the variance process \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(V_{t})_{t \ge 0}$$\end{document} is defined exogenously under Assumption 4.1. We in fact consider the slightly more general BSPDEs

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{array}{r@{\;}l} -\textrm{d}u(t,x) & = \displaystyle \bigg \{\frac{V_{t}}{2} \textrm{D}_{x}^{2} u(t,x)+\rho \sqrt{V_{t}} \textrm{D}_{x}\psi (t,x)-\frac{V_{t}}{2} \textrm{D}_{x}u(t,x) \\ & \quad \displaystyle + f\left( t, \textrm{e}^{x}, u(t,x), \rho _2\sqrt{V_{t}} \textrm{D}_{x}u(t,x), \psi (t,x)+ \rho _1 \sqrt{V_{t}} \textrm{D}_{x}u(t,x)\right) \bigg \} \textrm{d}t \\ & \quad \displaystyle -\psi (t,x) \textrm{d}W^1_{t}, \quad (t, x) \in [0, T) \times \mathbb {R}, \\ u(T,x) & = \displaystyle g \left( \textrm{e}^{x}\right) , \quad x \in \mathbb {R}. \end{array} \end{aligned}$$\end{document}

The following assumption on f and g (from [6]) ensures well-posedness of the above BSPDE, and we shall additionally require the existence of a weak-Sobolev solution (Assumption 5.1 ) for the convergence analysis of our numerical scheme in Section 5.

Assumption 4.2

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g:\mathbb {R}\rightarrow \mathbb {R}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f:[0,T]\times \mathbb {R}^4\rightarrow \mathbb {R}$$\end{document} be such that

• (i)g admits at most linear growth;
• (ii)f is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_f$$\end{document} -Lipschitz in all space arguments and there exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_0>0$$\end{document} such that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ |f(t,x,0,0,0)| \le L_f(1{+}|x|) \quad {\text {and}}\quad |f(t,x,y,z,\widetilde{z})-f(t,x,y,0,0)| \le L_0. $$\end{document}

Note that (4.2) is just a particular case of the general BSPDE (4.3) for the choice \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f(t, x, y, z, \widetilde{z}) \equiv -r y$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g(\textrm{e}^{x})\equiv h(\textrm{e}^{x+r T})$$\end{document} . Again, this general form is shown to be well posed in the distributional sense under Assumption 4.2 (borrowed from [6]). By [14] the corresponding BSDE is then, for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0 \le t \le s < T$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{aligned} -\textrm{d}Y_{s}^{t, x}&=f\left( s, \textrm{e}^{X_{s}^{t, x}}, Y_{s}^{t, x}, {Z_{s}^{1}}^{t, x}, {Z_{s}^{2}}^{t, x}\right) \textrm{d}s - {Z_{s}^{1}}^{t, x} \textrm{d}W^1_{s} - {Z_{s}^{2}}^{t, x} \textrm{d}W^2_{s},\\ Y_{T}^{t, x}&=g\left( \textrm{e}^{X_{T}^{t, x}}\right) , \end{aligned}\right. \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(Y_{s}^{t, x}, {Z_{s}^{1}}^{t, x}, {Z_{s}^{2}}^{t, x})$$\end{document} is defined as the solution to (4.4) in the weak sense.

Random Neural Network Scheme

Let the quadruple \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( X_{s}, Y_{s}, Z^1_{s}, Z^2_{s}\right) $$\end{document} be the solution to the forward FBSDE

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{aligned} -\textrm{d}Y_{s}&=f\left( s, \textrm{e}^{X_{s}}, Y_{s}, Z^1_{s}, Z^2_{s}\right) \textrm{d}s- Z^1_{s} \textrm{d}W^1_{s}-Z^2_{s} \textrm{d}W^2_{s},\\ \textrm{d}X_{s}&=-\frac{V_{s}}{2} \textrm{d}s+\sqrt{V_{s}}\left( \rho _1 \textrm{d}W^1_{s} + \rho _2\textrm{d}W^2_{s}\right) ,\\ V_{s}&= \xi _{s} \mathcal {E}\left( \eta \widehat{W}_{s}\right) , \quad \text{ with } \quad \widehat{W}_{s}=\int _{0}^{s} \mathcal {K}(s, r) \textrm{d}W^1_{r}, \end{aligned}\right. \end{aligned}$$\end{document}

for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s \in [0, T)$$\end{document} , with terminal condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_{T} = g\left( \textrm{e}^{X_{T}}\right) $$\end{document} , initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{0} = x$$\end{document} and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {K}$$\end{document} a locally square-integrable kernel and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _s:=\xi (s)>0$$\end{document} is the forward variance curve. For notational convenience below, we use \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _2 :=\sqrt{1-\rho _1^2}$$\end{document} , with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _1 \in [-1,1]$$\end{document} . Here  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {E}(\cdot )$$\end{document} denotes the Wick stochastic exponential and is defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {E}(\zeta ):=\exp \left\{ \zeta -\frac{1}{2}\mathbb {E}[|\zeta |^2]\right\} $$\end{document} for a centered Gaussian variable \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta $$\end{document} . Then by [6, Theorem 2.4],

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_{t}&= u\left( t, X_{t}\right) ,&{\text { for }} t\in [0,T], \\ Z^1_{t}&= \psi \left( t, X_{t}\right) + \rho _1 \sqrt{V_{t}} \textrm{D}_{x}u\left( t, X_{t}\right) ,&for t\in [0,T), \\ Z^2_{t}&= \rho _2\sqrt{V_{t}} \textrm{D}_{x}u\left( t, X_{t}\right) ,&for t\in [0,T), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(u, \psi )$$\end{document} is the unique weak solution to (4.3). Accordingly, the forward equation reads

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_t = Y_0-\int _{0}^{t} f\left( s, \textrm{e}^{X_{s}}, Y_s, Z_s^1, Z_s^2\right) \textrm{d}s +\int _{0}^{t}Z_s^1 \textrm{d}W^1_{s}+\int _{0}^{t} Z_s^2 \textrm{d}W^2_{s}, \quad {\text { for }}t \in [0, T]. \end{aligned}$$\end{document}

By simulating \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(W^1, W^2, V)$$\end{document} , the forward process X may be approximated by an Euler scheme–with the same notations as in the Markovian case–and the forward representation above yields the approximation

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ u\left( t_{i+1}, X_{t_{i+1}}\right) \approx u(t_{i}, X_{t_i}) - f\left( t_i, \textrm{e}^{X_{t_{i}}}, u\left( {t_{i}}, X_{t_{i}}\right) , Z^1_{t_i}, Z^2_{t_i}\right) \delta _{i}+ Z^1_{t_i} \Delta ^{W^1}_{i} + Z^2_{t_i} \Delta ^{W^2}_{i}, $$\end{document}

with

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ Z^1_{t_i} = \rho _1 \sqrt{V_{t_{i}}} \textrm{D}_{x}u\left( {t_{i}}, X_{t_{i}}\right) + \psi \left( {t_{i}}, X_{t_{i}}\right) \qquad {\text {and}}\qquad Z^2_{t_i} = \rho _2\sqrt{V_{t_{i}}} \textrm{D}_{x}u\left( {t_{i}}, X_{t_{i}}\right) . $$\end{document}

By Lemma A.2 we can, for each time step \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{0, \dots , N-1\}$$\end{document} , approximate the solutions  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$u(t_i,\cdot )$$\end{document} and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi (t_i,\cdot )$$\end{document} by two separate networks  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathfrak {U}_i$$\end{document} and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Psi _i$$\end{document} in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\aleph _K^{{{\,\mathrm{\varsigma }\,}}}$$\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_{t_i}&\approx \mathfrak {U}_i(X_{t_i}; \Theta ^i) = \Theta ^i \Phi ^{\Theta ,i}_K(X_{t_i}), \\ Z^1_{t_i}&\approx \mathfrak {Z}^1_i(X_{t_i}; \Theta ^i, \Xi ^i) = \Theta ^i \left( \textrm{D}_{x}\Phi _K^{\Theta ,i}(X_{t_i})\right) \rho _1\sqrt{V_{t_i}} + \Xi ^i\Phi ^{\Xi ,i}_K(X_{t_i}), \\ Z^2_{t_i}&\approx \mathfrak {Z}^2_i(X_{t_i}; \Theta ^i, \Xi ^i) = \Theta ^i \left( \textrm{D}_{x}\Phi _K^{\Theta ,i}(X_{t_i})\right) \rho _2\sqrt{V_{t_i}}. \end{aligned}$$\end{document}

Here \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi _K^\Xi $$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi _K^\Theta $$\end{document} are realisations of random bases (reservoirs) of the RWNNs with respective parameters  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Xi $$\end{document} and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Theta $$\end{document} . The next part relies on Assumption 3.2, namely

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ f\left( t_i, \textrm{e}^{X_{t_{i}}}, Y_{t_i}, Z^1_{t_i}, Z^2_{t_i}\right) = a(t_i,X_{t_i}) Y_{t_i} + b(t_i,X_{t_i}) Z^1_{t_i} + c(t_i,X_{t_i}) Z^2_{t_i} + \widetilde{f}(t_i,X_{t_i}), $$\end{document}

for some functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a,b,c,\widetilde{f}$$\end{document} mapping to  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}$$\end{document} , so that, as in the Markovian case, the minimisation of the expected quadratic loss at every time step \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{N-1,\dots ,0\}$$\end{document} reads

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\ell (\Theta ^i, \Xi ^i) \\ &:=\mathbb {E}^\Phi \Bigg [\bigg |{\widehat{\mathfrak {U}}}_{i+1}(X_{t_{i+1}})-\bigg \{\mathfrak {U}_i(X_{t_i}; \Theta ^i)-f\Big (t_{i}, X_{t_{i}}, \mathfrak {U}_i(X_{t_i}; \Theta ^i), \mathfrak {Z}^1_i(X_{t_i}; \Theta ^i, \Xi ^i), \mathfrak {Z}^2_i(X_{t_i}; \Theta ^i, \Xi ^i)\Big )\delta _{i}\\&\hspace{3cm} + \sum _{k=1}^2{\widehat{\mathfrak {Z}}}^k_i(X_{t_i}; \Theta ^i, \Xi ^i)\Delta ^{W^k}_{i}\bigg \}\bigg |^2\Bigg ] \\&= \mathbb {E}^\Phi \Bigg [\bigg |{\widehat{\mathfrak {U}}}_{i+1}(X_{t_{i+1}})-\bigg \{\mathfrak {U}_i(X_{t_i}; \Theta ^i)-\left( a\mathfrak {U}_i(X_{t_i}; \Theta ^i) + b\mathfrak {Z}^1_i(X_{t_i}; \Theta ^i, \Xi ^i) + c\mathfrak {Z}^2_i(X_{t_i}; \Theta ^i, \Xi ^i) + \widetilde{f}\right) \delta _{i}\\&\hspace{3cm} + \sum _{k=1}^2\mathfrak {Z}^k_i(X_{t_i}; \Theta ^i, \Xi ^i)\Delta ^{W^k}_{i}\bigg \}\bigg |^2\Bigg ] \\&= \mathbb {E}^\Phi \Bigg [\bigg |{\widehat{\mathfrak {U}}}_{i+1}(X_{t_{i+1}}) + \widetilde{f}\delta _{i}- \bigg \{ \Xi ^i\Phi _K^{\Xi ,i}(X_{t_i})\left( \Delta ^{W^1}_{i}-b\delta _{i}\right) \\&\hspace{3cm} + \Theta ^i\left( (1{-}a\delta _{i})\Phi _K^{\Theta ,i}(X_{t_i}) {+} \textrm{D}_{x}\Phi _K^{\Theta ,i}(X_{t_i})\sqrt{V_{t_i}}\left( \Delta ^{B}_{i} - (b\rho _1 {+} c\rho _2)\delta _{i}\right) \right) \bigg \}\bigg |^2\Bigg ] \\&= \mathbb {E}^\Phi \left[ \left| \textrm{Y}^i - \Xi ^i \textrm{X}_1^i - \Theta ^i\textrm{X}_2^i \right| ^2\right] , \end{aligned}$$\end{document}

with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta ^{B}_{i}=(\rho _1\Delta ^{W^1}_{i} + \rho _2\Delta ^{W^2}_{i})$$\end{document} and where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{\mathfrak {U}}}_{i+1}(X_{t_{i+1}}):=\mathfrak {U}_{i+1}(X_{t_{i+1}}; \Theta ^{i+1,*})$$\end{document} was set in the previous time step and is now constant (without dependence on  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Theta ^i$$\end{document} ). We defined

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{array}{r@{\;}l} \textrm{Y}^i & :=\displaystyle {\widehat{\mathfrak {U}}}_{i+1}(X_{t_{i+1}}) + \widetilde{f}\delta _{i}, \\ \textrm{X}_1^i & :=\displaystyle \Phi _K^\Xi (X_{t_i})\left( \Delta ^{W^1}_{i}-b\delta _{i}\right) , \\ \textrm{X}_2^i & :=\displaystyle (1-a\delta _{i})\Phi _K^\Theta (X_{t_i}) + \textrm{D}_{x}\Phi _K^\Theta (X_{t_i}) \sqrt{V_{t_i}}\left( \Delta ^{B}_{i}-(b\rho _1 + c\rho _2)\delta _{i}\right) . \end{array} \right. \end{aligned}$$\end{document}

In matrix form, this yields \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \ell (\Theta ^i, \Xi ^i) = \mathbb {E}^\Phi [\Vert \textrm{Y}^i - \boldsymbol{\beta }^i\textrm{X}^i\Vert ^2]$$\end{document} , with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\beta }^i=\begin{bmatrix}\Xi ^i, \Theta ^i\end{bmatrix}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{X}^i= \begin{bmatrix}\textrm{X}_1^i, \textrm{X}_2^i\end{bmatrix}^{\top }$$\end{document} , for which the RLS from Section 2.2 yields the solution

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \boldsymbol{\beta }^i = \mathbb {E}^\Phi \left[ \left[ \textrm{Y}^i\textrm{X}_1^{i\top } \quad \textrm{Y}^i\textrm{X}_2^{i\top }\right] \right] \mathbb {E}^\Phi \left[ \left[ \begin{array}{cc} \textrm{X}_1^i\textrm{X}_1^{i\top } & \textrm{X}_1^i\textrm{X}_2^{i\top } \\ \textrm{X}_2^i\textrm{X}_1^{i\top } & \textrm{X}_2^i\textrm{X}_2^{i\top } \end{array}\right] \right] ^{-1}. \end{aligned}$$\end{document}

Algorithm

We summarise the steps of the algorithm below: Algorithm 2RWNN non-Markovian scheme

Convergence Analysis

In this section, whenever there is any ambiguity, we use the notation  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X^\pi $$\end{document} to denote the discretised version of the solution process of (4.5) over the partition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi =\left\{ 0=t_{0}<t_{1}<\ldots <t_{N}=T\right\} $$\end{document} of the interval [0, T], with modulus \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\pi |=\max _{i\in \{0,1,\dots , N-1\}} \delta _{i}$$\end{document} with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta _{i}=t_{i+1}-t_{i}$$\end{document} . As mentioned just before Assumption 3.2, the linearity of f assumed before was only required to cast the optimisation in Algorithm 2 into a regression problem. In the forthcoming convergence analysis, this does not play any role, and we therefore allow for a more general function f.

Assumption 5.1

• (i)There exists a unique weak solution to the BSPDE system (4.3) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$u, \psi \in \mathcal {W}^{3,2}$$\end{document} ;
• (ii)There is an increasing continuous function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega :\mathbb {R}^+ \rightarrow \mathbb {R}^+$$\end{document} with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega (0)=0$$\end{document} such that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathbb {E}\left[ \int _{t_1}^{t_2}V_s\textrm{d}s\right] + \mathbb {E}\left[ \left| \int _{t_1}^{t_2}V_s\textrm{d}s\right| ^2\right] \le \omega (|t_2-t_1|), \quad {\text {for any }}0\le t_1 \le t_2 \le T; $$\end{document}

• (iii)There exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_f>0$$\end{document} such that, for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(t,x,z^1,z^2)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\widetilde{t},\widetilde{x},\widetilde{z}^1,\widetilde{z}^2)$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\left| f\left( t, \textrm{e}^x, y, z^1, z^2\right) -f\left( \widetilde{t}, \textrm{e}^{\widetilde{x}},\widetilde{y},\widetilde{z}^1,\widetilde{z}^2\right) \right| \\&\hspace{1cm} \le L_f\left\{ \omega (|t-\widetilde{t}|)^{\frac{1}{2}} + |x-\widetilde{x}| + |y-{\widetilde{y}}| + |z^1-\widetilde{z}^1| + |z^2-\widetilde{z}^2|\right\} . \end{aligned}$$\end{document}

Remark 5.2

Assumption 5.1(iii) may look unusual but appears as we are interested in evaluating options on the stock price, which is the exponential of the log stock price. This condition (the same as in [6]) allows us to control the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L^2$$\end{document} norm of the drift term in (4.5) (see also [13, Assumption 2.12(iii)]) and thus of the backward process Y therein.

Assumption 5.3

Given the partition  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi $$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \sup _{i\in \{0,\dots , N-1\}}\mathbb {E}\left[ \left| V^\pi _{t_i}\right| \right] $$\end{document} is finite.

Classical estimates (see Lemma C.1 for full details and proof) yield

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {E}\left[ \sup _{0 \le t \le T}\left| X_{t}\right| ^{2}\right] \le C\left( 1+\left| x_{0}\right| ^{2}\right) , \end{aligned}$$\end{document}

as well as (Lemma C.2)

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \max _{i\in \{0,\dots ,N-1\}} \mathbb {E}\left[ \left| X_{t_{i+1}}-X^\pi _{t_{i+1}}\right| ^{2}+\sup _{t \in \left[ t_{i}, t_{i+1}\right] }\left| X_{t}-X^\pi _{t_{i}}\right| ^{2}\right] \le C \omega (|\pi |), \end{aligned}$$\end{document}

for some \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C>0$$\end{document} independent of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\pi |$$\end{document} , and we furthermore have [14]

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {E}\left[ \int _{0}^{T}\left| f\left( t, \textrm{e}^{X_{t}}, Y_{t}, Z^1_{t}, Z^2_{t}\right) \right| ^{2} \textrm{d}t\right] <\infty , \end{aligned}$$\end{document}

as well as the standard \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L^{2}$$\end{document} -regularity result on Y:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \max _{i\in \{0, \dots , N-1\}} \mathbb {E}\left[ \sup _{t \in \left[ t_{i}, t_{i+1}\right] }\left| Y_{t}-Y^\pi _{t_{i}}\right| ^{2}\right] = \mathcal {O}(|\pi |). \end{aligned}$$\end{document}

For \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k\in \{1,2\}$$\end{document} , define the errors

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \varepsilon ^{Z^k}(\pi ) :=\mathbb {E}\left[ \sum _{i=0}^{N-1} \int _{t_{i}}^{t_{i+1}}\left| Z^k_{t}-\overline{Z}^k_{t_{i}}\right| ^{2} \textrm{d}t\right] , \quad \text{ with } \quad \overline{Z}^k_{t_{i}} :=\frac{1}{\delta _{i}} \mathbb {E}_{i}\left[ \int _{t_{i}}^{t_{i+1}} Z^k_{t} \textrm{d}t\right] , \end{aligned}$$\end{document}

which represent measures of the total variance of the processes  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z^k$$\end{document} along the partition  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi $$\end{document} and where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {E}_{i}$$\end{document} denotes the conditional expectation given \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {F}_{t_{i}}$$\end{document} . We furthermore define the auxiliary processes, for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{0, \ldots , {N-1}\}$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \widehat{\mathcal {V}}_{t_{i}}&:=\mathbb {E}_{i}\left[ \widehat{\mathfrak {U}}_{i+1}\left( X^\pi _{t_{i+1}}\right) \right] +f\left( t_i, \textrm{e}^{X^\pi _{t_{i}}}, \widehat{\mathcal {V}}_{t_{i}}, \overline{\widehat{Z}}^{1}_{t_{i}}, \overline{\widehat{Z}}^{2}_{t_{i}}\right) \delta _{i}, \\ \overline{\widehat{Z}}^{1}_{t_{i}}&:={\widehat{\Psi }}_i(X^\pi _{t_i}) + \frac{1}{\delta _{i}} \mathbb {E}_{i}\left[ \widehat{\mathfrak {U}}_{i+1}\left( X^\pi _{t_{i+1}}\right) \Delta ^{W^1}_{i}\right] , \\ \overline{\widehat{Z}}^{2}_{t_{i}}&:=\frac{1}{\delta _{i}} \mathbb {E}_{i}\left[ \widehat{\mathfrak {U}}_{i+1}\left( X^\pi _{t_{i+1}}\right) \Delta ^{W^2}_{i}\right] , \end{aligned} \end{aligned}$$\end{document}

with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{\mathfrak {U}}}_i(\textbf{x}):=\mathfrak {U}_i(x;\Theta ^{i,*})$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ {\widehat{\Psi }}_i(x):=\Psi (x;\Xi ^{i,*})$$\end{document} as before. Observe that  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\Psi }_{i+1}$$\end{document} and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\mathfrak {U}}_{i+1}$$\end{document} do not depend on  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Theta ^i$$\end{document} because the parameters were fixed at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(i+1)$$\end{document} time step and are held constant at step i (see Algorithm 2). Next, notice that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\mathcal {V}}$$\end{document} is well defined by a fixed-point argument since f is Lipschitz. By Assumption 5.1(i), there exist \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{v}}_i, \overline{\widehat{z}}^{k}_{t_{i}}$$\end{document} for which

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\widehat{\mathcal {V}}}_{t_i} = {\widehat{v}}_i(X^\pi _{t_i})\qquad {\text {and}} \qquad \overline{\widehat{Z}}^{k}_{t_{i}} = \overline{\widehat{z}}^{k}_{t_{i}}(X^\pi _{t_i}) \quad {\text {for }} i\in \{0,\dots ,N-1\}, k\in \{1,2\}. \end{aligned}$$\end{document}

By the martingale representation theorem, one has integrable processes  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\widehat{Z}}^{1}, \overline{\widehat{Z}}^{2}$$\end{document} such that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \widehat{\mathfrak {U}}_{i+1}\left( X^\pi _{t_{i+1}}\right) =\widehat{\mathcal {V}}_{t_{i}}-f\left( t_{i},\textrm{e}^{X^\pi _{t_{i}}}, \widehat{\mathcal {V}}_{t_{i}}, \overline{\widehat{Z}}^{1}_{t_{i}}, \overline{\widehat{Z}}^{2}_{t_{i}}\right) \delta _{i}+\int _{t_{i}}^{t_{i+1}}{\widehat{Z}}_{t}^1\textrm{d}W^1_{t}+\int _{t_{i}}^{t_{i+1}} {\widehat{Z}}_{t}^2 \textrm{d}W^2_{t}, \end{aligned}$$\end{document}

since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\widehat{Z}}^{k}_{t}$$\end{document} are \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {F}_t^W$$\end{document} -adapted as asserted by the martingale representation theorem. From here, Itô’s isometry yields

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \overline{\widehat{Z}}^{1}_{t_{i}}&= {\widehat{\Psi }}_i(X^\pi _{t_i}) + \frac{1}{\delta _{i}}\mathbb {E}_i\left[ {\widehat{\mathfrak {U}}}_{i+1}(X^\pi _{t_{i+1}})\Delta ^{W^k}_{i} \right] \\&= \frac{1}{\delta _{i}}\int _{t_i}^{t_{i+1}}{\widehat{\Psi }}_i(X^\pi _{t_i})\textrm{d}t + \frac{1}{\delta _{i}}\mathbb {E}_i\left[ \left( {\widehat{\mathcal {V}}}_{t_i} +\int _{t_{i}}^{t_{i+1}} \widehat{Z}_{t}^1 \textrm{d}W^1_{t}+\int _{t_{i}}^{t_{i+1}} \widehat{Z}_{t}^2 \textrm{d}W^2_{t} \right) \int _{t_i}^{t_{i+1}} \textrm{d}W^1_t \right] \\&= \frac{1}{\delta _{i}}\mathbb {E}_i\left[ \int _{t_i}^{t_{i+1}}\left( {\widehat{\Psi }}_i(X^\pi _{t_i}) + \widehat{Z}_{t}^1 \right) \textrm{d}t \right] , \end{aligned}$$\end{document}

and similarly,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \overline{\widehat{Z}}^{2}_{t_{i}} = \frac{1}{\delta _{i}}\mathbb {E}_i\left[ \int _{t_i}^{t_{i+1}} \widehat{Z}_{t}^2 \textrm{d}t \right] . $$\end{document}

We consider convergence in terms of the following error:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \mathscr {E}\left( \widehat{\mathfrak {U}}, {\widehat{\Psi }}\right) :=&\max _{i\in \{0, \dots , N-1\}} \mathbb {E}^\Phi \left[ \left| Y_{t_i}-\widehat{\mathfrak {U}}_i\left( X^\pi _{t_i}\right) \right| ^2\right] +\mathbb {E}^\Phi \left[ \sum _{i=0}^{N-1} \int _{t_i}^{t_{i+1}}\sum _{k=1}^2\left| Z_t^k-\widehat{\mathcal {Z}}^k_i\left( X^\pi _{t_i}\right) \right| ^2 \textrm{d}t\right] , \end{aligned} \end{aligned}$$\end{document}

with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\mathcal {Z}}^1_{i}, \widehat{\mathcal {Z}}^2_{i}$$\end{document} introduced before Lemma 5.8. We now state the main convergence result:

Theorem 5.4

Under Assumptions 4.1-4.2-5.1, there exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C>0$$\end{document} such that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathscr {E}\left( \widehat{\mathfrak {U}}, {\widehat{\Psi }}\right) \le C\left\{ \omega (|\pi |) {+} \mathbb {E}\left[ \left| g(X_T)-g(X^{\pi }_{T})\right| ^2\right] {+}\sum _{k=1}^2\varepsilon ^{Z^k}(\pi ){+}\frac{C^{*}}{K}N{+} M|\pi |^2\right\} , $$\end{document}

with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C^*, M>0$$\end{document} given in Lemma 5.9 and the errors  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varepsilon ^{Z^k}(\pi )$$\end{document} defined in (5.4).

The following follows from (B.5), established in Part II of the proof of Theorem 5.4:

Corollary 5.5

Under Assumptions 4.1-4.2-5.1, there exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C>0$$\end{document} such that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \max _{i\in \{0, \dots , N-1\}}&\mathbb {E}^\Phi \left[ \left| Y_{t_i}-\widehat{\mathfrak {U}}_i\left( X^\pi _{t_i}\right) \right| ^2\right] \le \\ &C\left\{ \omega (|\pi |) + \mathbb {E}\left[ \left| g(X_T)-g(X^{\pi }_{T})\right| ^2\right] +\sum _{k=1}^2\varepsilon ^{Z^k}(\pi )+\frac{C^{*}}{K}N+ M|\pi |^2\right\} , \end{aligned}$$\end{document}

with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C^*, M>0$$\end{document} given in Lemma 5.9.

Remark 5.6

The second error term is the strong \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L^2$$\end{document} -Monte-Carlo error and is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(N^{-H})$$\end{document} for processes driven by an fBm with Hurst parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H\in (0, 1)$$\end{document} . We refer the reader to [13, 26] for an exposition on strong versus weak error rates in rough volatility models.

To prove Theorem 5.4, the following bounds on the derivatives are key.

Lemma 5.7

Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Psi _K(\cdot ; \Theta )\in \aleph ^{{{\,\mathrm{\varsigma }\,}}}_K$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(X^\pi _{t_i}, V^\pi _{t_i})_{i}$$\end{document} denote the discretised versions of (4.5) over the partition  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi $$\end{document} , then there exist \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_1, L_2 > 0$$\end{document} such that, for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{0,\dots ,N-1\}$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \left\| \mathbb {E}_i^\Phi \left[ \textrm{D}_{x}\Psi _K(X^\pi _{t_{i+1}}; \Theta )\right] \right\| \le L_1 \qquad {\text {and}} \qquad \left\| \mathbb {E}_i^\Phi \left[ \textrm{D}_{x}^2\Psi _K(X^\pi _{t_{i+1}}; \Theta )\right] \right\| \le L_2. $$\end{document}

Proof

We start with the first derivative. For all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x,y\in \mathbb {R}^d$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\| \Psi _K(x; \Theta )-\Psi _K(y; \Theta )\right\|&= \left\| \Theta \left( {{\,\mathrm{\boldsymbol{\varsigma }}\,}}(\textrm{A}x+b) -{{\,\mathrm{\boldsymbol{\varsigma }}\,}}(Ay+b)\right) \right\| \le \Vert \Theta \Vert _F\Vert {{\,\mathrm{\boldsymbol{\varsigma }}\,}}(\textrm{A}x+b)\\&\quad -{{\,\mathrm{\boldsymbol{\varsigma }}\,}}(Ay+b)\Vert \\&{\le } \Vert \Theta \Vert _F\Vert \textrm{A}x-Ay\Vert \le \Vert \Theta \Vert _F\Vert A\Vert _F\Vert x{-}y\Vert \le L_1\Vert x-y\Vert , \end{aligned}$$\end{document}

since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\varsigma }\,}}$$\end{document} is 1-Lipschitz. The estimator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Theta $$\end{document} has an explicit form (4.7) and its norm is finite, therefore \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Psi _K(\cdot ; \Theta )$$\end{document} is globally Lipschitz and its first derivative is bounded by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_1>0$$\end{document} . Next, without loss of generality, we can set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{A}=\textrm{I}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{b}=0$$\end{document} , since their support is bounded. As in (2.2) for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\in \{1,\dots ,m\}$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {E}_i^\Phi \left[ \textrm{D}_{x}^2 \Psi _K(X^\pi _{t_{i+1}}; \Theta )\right] _j&= \int \Theta \operatorname {diag}\left( e_{j}\right) \operatorname {diag}\left( \boldsymbol{\delta }_0\left( x-\frac{1}{2} V^\pi _{t_i}\delta _{i}+ \sqrt{V_{t_i}}w\right) \right) {\boldsymbol{p}}_\mathcal {N}(w) \textrm{d}w \\&= \Theta \operatorname {diag}\left( e_{j}\right) \operatorname {diag}\left( {\boldsymbol{p}}_\mathcal {N}\left( 0; x-\frac{1}{2} V^\pi _{t_i}\delta _{i}, V^\pi _{t_i}\delta _{i}\right) \right) , \end{aligned}$$\end{document}

since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta ^{B}_{i}\sim \mathcal {N}(0, \delta _{i})$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\boldsymbol{p}}_\mathcal {N}$$\end{document} is the Gaussian density applied component-wise. Since the weights are sampled on a compact and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Vert \Theta \Vert $$\end{document} is finite, then there exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C>0$$\end{document} such that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \left\| \mathbb {E}_i^\Phi \left[ \textrm{D}_{x}^2 \Psi _K(X^\pi _{t_{i+1}}; \Theta )\right] \right\| \le C\Vert \Theta \Vert _F = L_2. $$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

From here the error bound of approximating \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{\mathcal {V}}}_{t_i}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\widehat{Z}}^{1}_{t_{i}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\widehat{Z}}^{2}_{t_{i}}$$\end{document} with their RWNN approximators \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{\mathfrak {U}}}_i$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\mathcal {Z}}^1_{i}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\mathcal {Z}}^2_{i}$$\end{document} (defined in the lemma below) can be obtained. For \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{0,\dots ,N-1\}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\mathfrak {U}_i, \Psi _i)\in \aleph ^{{{\,\mathrm{\varsigma }\,}}}_K$$\end{document} , introduce

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{array}{r@{\;}lr@{\;}l} \mathcal {Z}^1_{i}(x) & :=\Psi _i(x)+\rho _1\sqrt{V_{t_i}}\textrm{D}_{x}\mathfrak {U}_i(x), & \qquad \mathcal {Z}^2_{i}(x) & :=\rho _2\sqrt{V_{t_i}}\textrm{D}_{x}\mathfrak {U}_i(x),\\ \widehat{\mathcal {Z}}^1_{i}(x) & :={\widehat{\Psi }}_i(x)+\rho _1\sqrt{V_{t_i}}\textrm{D}_{x}{\widehat{\mathfrak {U}}}_i(x), & \qquad \widehat{\mathcal {Z}}^2_{i}(x) & :=\rho _2\sqrt{V_{t_i}}\textrm{D}_{x}{\widehat{\mathfrak {U}}}_i(x). \end{array} \end{aligned}$$\end{document}

Lemma 5.8

Under Assumptions 5.1-5.3, there exists \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M>0$$\end{document} such that

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathbb {E}^\Phi \left[ \left| \mathcal {Z}_i^k(X^\pi _{t_i}) - \overline{\widehat{Z}}^{k}_{t_{i}}\right| ^2\right] \le \rho _k^2|\pi |^2 M, \quad {\text {for all }}i\in \{0,\dots ,N-1\}, k=1,2. $$\end{document}

Proof

From (5.5) and (5.6), we have, for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{0,\dots ,N-1\}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k\in \{1,2\}$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \widehat{v}_{i}(x)&= \mathbb {E}^\Phi _{i}\left[ \widehat{\mathfrak {U}}_{i+1}\left( X^{x,\pi }_{t_{i+1}}\right) \right] + f\left( t_i, \textrm{e}^{x}, \widehat{v}_{i}(x), \overline{\widehat{z}}^{1}_{i}(x), \overline{\widehat{z}}^{2}_{i}(x)\right) \delta _{i}, \\ \overline{\widehat{z}}^{k}_{i}(x)&= \Psi _i\left( X^{x,\pi }_{t_{i+1}}\right) 1\hspace{-2.1mm}{1}_{\{k=1\}} + \frac{1}{\delta _{i}} \mathbb {E}^\Phi _{i}\left[ \widehat{\mathfrak {U}}_{i+1}\left( X^{x,\pi }_{t_{i+1}}\right) \Delta ^{W^k}_{i}\right] , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X^{x,\pi }_{t_{i+1}} = x + \left( r- \frac{1}{2} V_{t_i}\right) \delta _{i}+ \sqrt{V_{t_i}}\Delta ^{B}_{i}$$\end{document} is the Euler discretisation of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{X_t\}_{t\in [0, T]}$$\end{document} over  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi $$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{V^\pi _{t_i}\}_{i=0}^{N}$$\end{document} is the appropriate discretisation of the volatility process over the same partition. For \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{\mathcal {R}^k\}\overset{{\textrm{iid}}}{\sim }\mathcal {N}(0,1)$$\end{document} , the two auxiliary processes can be written as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \overline{\widehat{z}}^{k}_{i}(x)&= \Psi _i\left( X^{x,\pi }_{t_{i+1}}\right) 1\hspace{-2.1mm}{1}_{\{k=1\}} + \frac{1}{\delta _{i}} \mathbb {E}^\Phi _{i}\left[ \widehat{\mathfrak {U}}_{i+1}\left( x - \frac{V^{x,\pi }_{t_i}}{2}\delta _{i}+ \sqrt{V^{x,\pi }_{t_i}\delta _{i}}\left( \rho _1 \mathcal {R}^1 + \rho _2 \mathcal {R}^2\right) \right) \sqrt{\delta _{i}} \mathcal {R}^k\right] . \end{aligned}$$\end{document}

Notice that, while any sensible forward scheme for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{V_t\}_{t\in [0,T]}$$\end{document} does depend on a series of Brownian increments, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V^{x,\pi }_{t_i}$$\end{document} only depends on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( \Delta ^{W}_{i-1},\dots ,\Delta ^{W}_{0}\right) $$\end{document} , which are known at time  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_i$$\end{document} . Thus, since usual derivative operations are available for approximately differentiable functions (Remark 2.7) multivariate integration by parts for Gaussian measures (a formulation of Isserlis’ Theorem [44]) yields

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \overline{\widehat{z}}^{k}_{i}(x) = \Psi _i\left( X^{x,\pi }_{t_{i+1}}\right) 1\hspace{-2.1mm}{1}_{\{k=1\}} + \rho _k\sqrt{V_{t_i}}\mathbb {E}^\Phi \left[ \textrm{D}_{x}{\widehat{\mathfrak {U}}}_{i+1}\left( X^{x, \pi }_{t_{i+1}}\right) \right] , $$\end{document}

with corresponding derivatives

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \textrm{D}_{x}\overline{\widehat{z}}^{k}_{i}(x) = \textrm{D}_{x}\Psi _i\left( X^{x,\pi }_{t_{i+1}}\right) 1\hspace{-2.1mm}{1}_{\{k=1\}} + \rho _k\sqrt{V_{t_i}}\mathbb {E}^\Phi \left[ \textrm{D}_{x}^2 {\widehat{\mathfrak {U}}}_{i+1}\left( X^{x,\pi }_{t_{i+1}}\right) \right] . $$\end{document}

An application of the implicit function theorem then implies

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textrm{D}_{x}{\widehat{v}}_i(x) =&\mathbb {E}^\Phi _{i}\left[ \textrm{D}_{x}\widehat{\mathfrak {U}}_{i+1}\left( X^{x,\pi }_{t_{i+1}}\right) \right] \\&+ \delta _{i}\left\{ \textrm{D}_{x}{\widehat{f}}_i(x) + \textrm{D}_y {\widehat{f}}_i(x)\textrm{D}_{x}{\widehat{v}}_i(x) + \sum _{k=1}^{2}\textrm{D}_{z^k} {\widehat{f}}_i(x) \textrm{D}_{x}\overline{\widehat{z}}^{k}_{i}(x)\right\} , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{f}}_i(x) :=f\left( t_i, \textrm{e}^{x}, \widehat{v}_{i}(x), \overline{\widehat{z}}^{1}_{i}(x), \overline{\widehat{z}}^{2}_{i}(x)\right) $$\end{document} and

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\Psi _i\left( X^{x,\pi }_{t_{i+1}}\right) 1\hspace{-2.1mm}{1}_{\{k=1\}} + \left( 1-\delta _{i}\textrm{D}_y{\widehat{f}}_i(x)\right) \rho _k\sqrt{V^{\pi }_{t_i}}\textrm{D}_{x}{\widehat{v}}_i(x) \\&= \overline{\widehat{z}}^k_{i}(x) + \rho _k\sqrt{V^{\pi }_{t_i}}\delta _{i}\left( \textrm{D}_{x}{\widehat{f}}_i(x) + \textrm{D}_{z^1} {\widehat{f}}_i(x) \textrm{D}_{x}\overline{\widehat{z}}^{1}_{i}(x) + \textrm{D}_{z^2} {\widehat{f}}_i(x)\textrm{D}_{x}\overline{\widehat{z}}^{2}_{i}(x)\right) . \end{aligned}$$\end{document}

Thus, for small enough \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\pi |$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\Psi _i\left( X^{x,\pi }_{t_{i+1}}\right) 1\hspace{-2.1mm}{1}_{\{k=1\}} + \rho _k\sqrt{V^{\pi }_{t_i}}\textrm{D}_{x}{\widehat{v}}_i(x) \\&\qquad \le \overline{\widehat{z}}^k_{i}(x) + \rho _k\sqrt{V^{\pi }_{t_i}}\delta _{i}\left( \textrm{D}_{x}{\widehat{f}}_i(x) + \textrm{D}_{z^1} {\widehat{f}}_i(x) \textrm{D}_{x}\overline{\widehat{z}}^{1}_{i}(x) + \textrm{D}_{z^2} {\widehat{f}}_i(x)\textrm{D}_{x}\overline{\widehat{z}}^{2}_{i}(x)\right) , \end{aligned}$$\end{document}

and since f is Lipschitz by Assumption 5.1(iii), then \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{D}_{r} {\widehat{f}}_i(x)=1$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r\in \{x, z^1, z^2\}$$\end{document} and, by Lemma 5.7 and the definition of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\widehat{z}}^k_{i}(x)$$\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Psi _i\left( X^{x,\pi }_{t_{i+1}}\right) 1\hspace{-2.1mm}{1}_{\{k=1\}} + \rho _k\sqrt{V^{\pi }_{t_i}}\textrm{D}_{x}{\widehat{v}}_i(x)&\le \overline{\widehat{z}}^k_{i}(x) + \rho _k\delta _{i}\sqrt{V^{\pi }_{t_i}}\left( 1+\textrm{D}_{x}\overline{\widehat{z}}^{1}_{i}(x) + \textrm{D}_{x}\overline{\widehat{z}}^{2}_{i}(x)\right) \\&\le \overline{\widehat{z}}^k_{i}(x) + \rho _k\delta _{i}\sqrt{V^{\pi }_{t_i}}\left( 1 + L_1 + \sqrt{2} L_2 \sqrt{V^{\pi }_{t_i}}\right) . \end{aligned}$$\end{document}

Therefore, using the above inequality

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\mathbb {E}^\Phi \left[ \left| \Psi _i(X^\pi _{t_i}) + \rho _1\sqrt{V^\pi _{t_i}}\textrm{D}_{x}\mathfrak {U}_i(X^\pi _{t_i}) - \overline{\widehat{Z}}^{1}_{t_i}\right| ^2 \right] \\&\le \mathbb {E}^\Phi \left[ \left| \overline{\widehat{z}}^{1}_{i}(X^\pi _{t_i}) - \overline{\widehat{Z}}^{1}_{t_i} + \rho _1\delta _{i}\sqrt{V^\pi _{t_i}}\left[ 1 + L_1 + \sqrt{2} L_2 \sqrt{V^\pi _{t_i}}\right] \right| ^2 \right] \\&\le |\rho _1\delta _{i}|^2 \mathbb {E}\left[ \left| \sqrt{V^\pi _{t_i}}\left( 1+L_1+\sqrt{2}L_2\sqrt{V^\pi _{t_i}}\right) \right| ^2 \right] \\&\le \rho _1^2|\pi |^2 \left\{ \mathbb {E}\left[ \left| V^\pi _{t_i}\right| \right] \left( 1 + L_1 + \sqrt{2}L_2\mathbb {E}\left[ \left| V^\pi _{t_i}\right| \right] \right) \right\} \le \rho _1^2|\pi |^2 M, \end{aligned}$$\end{document}

relying on Corollary 2.10 and the fact that  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\widehat{Z}}^{k}_{t_{i}} = \overline{\widehat{z}}^{k}_{t_{i}}(X^\pi _{t_i})$$\end{document}  (see (5.6)) in the second inequality and the boundedness of  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {E}[|V^{\pi }|]$$\end{document} from Assumption 5.3 in the last line. The proof of the other bound is analogous. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Lemma 5.9

Under Assumptions 4.1-4.2-5.1, for sufficiently small  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\pi |$$\end{document} we have

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {E}^\Phi \left[ \left| \widehat{\mathcal {V}}_{t_{i}}-\widehat{\mathfrak {U}}_{i}\left( X^\pi _{t_{i}}\right) \right| ^{2}\right] +\delta _{i}\mathbb {E}^\Phi \left[ \sum _{k=1}^2\left| \widehat{{Z}}^k_{t_{i}}-\widehat{\mathcal {Z}}^k_{i}\left( X^\pi _{t_{i}}\right) \right| ^{2}\right] \le C\left\{ \frac{C^{*}}{K} + M|\pi |^3\right\} \end{aligned}$$\end{document}

for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{0, \dots , N-1\}$$\end{document} and K hidden units, for some \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C>0$$\end{document} , where  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C^*$$\end{document} is as in Proposition A.1 and M in Lemma 5.8.

Proof of Lemma 5.9

Fix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{0,\dots ,N-1\}$$\end{document} . Relying on the martingale representation in (5.7) and Lemma A.2, we can define the following loss function for the pair \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\mathfrak {U}_i(\cdot ; \Theta ),\Psi _i(\cdot ;\Xi ))\in \aleph ^{{{\,\mathrm{\varsigma }\,}}}_K$$\end{document} and their corresponding parameters  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Theta $$\end{document} and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Xi $$\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \widehat{L}_i(\Theta ,\Xi ) :=\widetilde{L}_i(\Theta ,\Xi ) + \mathbb {E}^\Phi \left[ \int _{t_i}^{t_{i+1}}\sum _{k=1}^2\left| {\widehat{Z}}_t^k - \overline{\widehat{Z}}^{k}_{t_{i}}\right| ^2\right] , \end{aligned}$$\end{document}

with

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\widetilde{L}_i(\Theta ,\Xi ) :=\mathbb {E}^\Phi \bigg [\Big |{\widehat{\mathcal {V}}}_{t_i} - \mathfrak {U}_i(X^\pi _{t_i}; \Theta )\\&\quad + \delta _{i}\Big \{ f\left( t_i, \textrm{e}^{X^\pi _{t_i}}, \mathfrak {U}(X^\pi _{t_i}; \Theta ), \mathcal {Z}^1_i(X^\pi _{t_i}; \Theta , \Xi ), \mathcal {Z}^2_i(X^\pi _{t_i}; \Theta , \Xi )\right) \\&\quad - f\left( t_i, \textrm{e}^{X^\pi _{t_i}}, \mathcal {V}_{t_i}, \overline{\widehat{Z}}^{1}_{t_{i}}, \overline{\widehat{Z}}^{2}_{t_{i}}\right) \Big \} \Big |^2\bigg ] \\&\quad + \delta _{i}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] . \end{aligned}$$\end{document}

Now, recall the following useful identity, valid for any \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a, b\in \mathbb {R}$$\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (a+b)^2 \le \left( 1+\chi \right) a^2 + \left( 1+\frac{1}{\chi }\right) b^2, \quad \chi > 0. \end{aligned}$$\end{document}

Applying (5.9) yields

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\widetilde{L}_i(\Theta , \Xi ) \le \delta _{i}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] \\ &\quad + (1 + C\delta _{i}) \mathbb {E}^\Phi \left[ \left\lvert {\widehat{\mathcal {V}}}_{t_i} - \mathfrak {U}_i(X^\pi _{t_i}; \Theta )\right\rvert ^2\right] \\&\quad + \left( 1+\frac{1}{C\delta _{i}}\right) \mathbb {E}^\Phi \left[ \left\lvert f\left( t_i, \textrm{e}^{X^\pi _{t_i}}, \mathfrak {U}(X^\pi _{t_i}; \Theta ), \mathcal {Z}^1_i(X^\pi _{t_i}; \Theta , \Xi ), \mathcal {Z}^2_i(X^\pi _{t_i}; \Theta , \Xi )\right) \right. \right. \\&\quad \left. \left. - f\left( t_i, \textrm{e}^{X^\pi _{t_i}}, \mathcal {V}_{t_i}, \overline{\widehat{Z}}^{1}_{t_{i}}, \overline{\widehat{Z}}^{2}_{t_{i}}\right) \right\rvert ^2\right] . \end{aligned}$$\end{document}

Now by the Lipschitz condition on f from Assumption 5.1,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \widetilde{L}_i(\Theta , \Xi ) \le (1 + C\delta _{i}) \mathbb {E}^\Phi \left[ \left\lvert {\widehat{\mathcal {V}}}_{t_i} - \mathfrak {U}_i(X^\pi _{t_i}; \Theta )\right\rvert ^2\right] + C\delta _{i}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] . \end{aligned}$$\end{document}

For any \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a, b\in \mathbb {R}$$\end{document} , inequality (5.9) holds with the reverse sign and hence also

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (a+b)^2 \ge \left( 1-\chi \right) a^2 - \frac{1}{\chi }b^2, \quad \chi > 0. \end{aligned}$$\end{document}

Following (5.10), for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi =\gamma \delta _{i}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma >0$$\end{document} we have

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\widetilde{L}_i(\Theta , \Xi ) \ge \delta _{i}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] + (1 - \gamma \delta _{i}) \mathbb {E}^\Phi \left[ \left\lvert {\widehat{\mathcal {V}}}_{t_i} - \mathfrak {U}_i(X^\pi _{t_i}; \Theta )\right\rvert ^2\right] \\&\quad - \frac{1}{\gamma \delta _{i}}\mathbb {E}^\Phi \left[ \left\lvert f\Big (t_i, \textrm{e}^{X^\pi _{t_i}}, \mathfrak {U}(X^\pi _{t_i}; \Theta ), \mathcal {Z}^1_i(X^\pi _{t_i}; \Theta , \Xi ), \mathcal {Z}^2_i(X^\pi _{t_i}; \Theta , \Xi )\Big )\right. \right. \\&\quad \left. \left. - f\left( t_i, \textrm{e}^{X^\pi _{t_i}}, \mathcal {V}_{t_i}, \overline{\widehat{Z}}^{1}_{t_{i}}, \overline{\widehat{Z}}^{2}_{t_{i}}\right) \right\rvert ^2\right] . \end{aligned}$$\end{document}

Again since f is Lipschitz, the arithmetic-geometric inequality implies

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \widetilde{L}_i(\Theta , \Xi )&\ge (1 - \gamma \delta _{i}) \mathbb {E}^\Phi \left[ \left\lvert {\widehat{\mathcal {V}}}_{t_i} - \mathfrak {U}_i(X^\pi _{t_i}; \Theta )\right\rvert ^2\right] + \delta _{i}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] \\&\qquad - \frac{\delta _{i}}{\gamma }\mathbb {E}^\Phi \left[ L_f^2 \bigg |\left\lvert {\widehat{\mathcal {V}}}_{t_i} - \mathfrak {U}(X^\pi _{t_i}; \Theta )\right\rvert + \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert \bigg |^2\right] \\&\ge (1 - \gamma \delta _{i}) \mathbb {E}^\Phi \left[ \left\lvert {\widehat{\mathcal {V}}}_{t_i} - \mathfrak {U}_i(X^\pi _{t_i}; \Theta )\right\rvert ^2\right] + \delta _{i}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] \\&\qquad - \frac{3\delta _{i}L_f^2}{\gamma }\left( \mathbb {E}^\Phi \left[ \left\lvert {\widehat{\mathcal {V}}}_{t_i} - \mathfrak {U}_i(X^\pi _{t_i}; \Theta )\right\rvert ^2\right] + \sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] \right) . \end{aligned}$$\end{document}

Taking \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =6L_f^2$$\end{document} gives

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \widetilde{L}_i(\Theta , \Xi ) \ge (1-C\delta _{i})\mathbb {E}^\Phi \left[ \left\lvert {\widehat{\mathcal {V}}}_{t_i} - \mathfrak {U}_i(X^\pi _{t_i}; \Theta )\right\rvert ^2\right] + \frac{\delta _{i}}{2}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] . \end{aligned}$$\end{document}

For a given \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{0,\dots ,N-1\}$$\end{document} take \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\Theta ^*, \Xi ^*)\in \mathop {\textrm{argmin}}\limits _{\Theta , \Xi } \widehat{L}_i(\Theta , \Xi )$$\end{document} so that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{\mathfrak {U}}}_i=\mathfrak {U}_i(\cdot ; \Theta ^*)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\mathcal {Z}}_i^k(\cdot ) :=\mathcal {Z}^k_i(\cdot ; \Theta ^*, \Xi ^*)$$\end{document} . From (5.8), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{L}_i$$\end{document} and  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widetilde{L}_i$$\end{document} have the same minimisers, thus combining both bounds gives for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\Theta ,\Xi )\in \mathbb {R}^{m\times K}\times \mathbb {R}^{m\times K}$$\end{document} ,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\left( 1-C \delta _{i}\right) \mathbb {E}^\Phi \left[ \left| {\widehat{\mathcal {V}}}_{t_{i}}-{\widehat{\mathfrak {U}}}_{i}\left( X^\pi _{t_{i}}\right) \right| ^{2}\right] +\frac{\delta _{i}}{2}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \widehat{\mathcal {Z}}_i^k\right\rvert ^2\right] \le \widetilde{L}_i(\Theta ^*,\Xi ^*)\le \widetilde{L}_{i}(\Theta , \Xi ) \\&\le \left( 1+C \delta _{i}\right) \mathbb {E}^\Phi \left[ \left| {\widehat{\mathcal {V}}}_{t_{i}}-\mathfrak {U}_{i}\left( X^\pi _{t_{i}} ; \Theta \right) \right| ^{2}\right] +C \delta _{i}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] . \end{aligned}$$\end{document}

Letting \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\pi |$$\end{document} be sufficiently small gives together with Lemma 5.8

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {E}^\Phi \left[ \left| {\widehat{\mathcal {V}}}_{t_{i}}-{\widehat{\mathfrak {U}}}_{i}\left( X^\pi _{t_{i}}\right) \right| ^{2}\right]&+ \delta _{i}\sum _{k=1}^2\mathbb {E}^\Phi \left[ \left\lvert \overline{\widehat{Z}}^{k}_{t_{i}} - \mathcal {Z}_i^k(X^\pi _{t_i}; \Theta , \Xi )\right\rvert ^2\right] \\&\le C \left\{ \inf _\Theta \mathbb {E}^\Phi \left[ \left| {\widehat{v}}_i(X^\pi _{t_i})-\mathfrak {U}_i(X^\pi _{t_i}; \Theta )\right| ^2\right] + |\pi |^3\left( \rho _1^2 + \rho _2^2\right) M\right\} , \end{aligned}$$\end{document}

therefore, using Proposition A.1, we obtain

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\mathbb {E}^\Phi \left[ \left| \widehat{\mathcal {V}}_{t_{i}}-\widehat{\mathfrak {U}}_{i}\left( X^\pi _{t_{i}}\right) \right| ^{2}\right] + \delta _{i}\mathbb {E}^\Phi \left[ \sum _{k=1}^2\left| \overline{\widehat{Z}}^{k}_{t_{i}}-\widehat{\mathcal {Z}}^k_{i}\left( X^\pi _{t_{i}}\right) \right| ^{2}\right] \\&\qquad \le C\left\{ \inf _\Theta \mathbb {E}^\Phi \left[ \left| {\widehat{v}}_i(X^\pi _{t_i})-\mathfrak {U}_i(X^\pi _{t_i}; \Theta )\right| ^2\right] + M|\pi |^3 \right\} \le C\left\{ \frac{C^{*}}{K} + M|\pi |^3 \right\} . \end{aligned}$$\end{document}

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

The rest of the proof is similar to those in [6, Theorem A.2] and [41, Theorem 4.1], but we include it in Appendix B for completeness.

Numerical Results

We now showcase the performance of the RWNN scheme on a representative model from each–Markovian and non-Markovian–class. We first test our scheme in the multidimensional Black-Scholes (BS) setting [12] and then move to the non-Markovian setup with the rough Bergomi (rBergomi) model [4]. We develop numerical approximations to European option prices given in (3.3) and (4.2), choosing

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ f(t, x, y, z^1, z^2) = -r y \qquad \text{ and } \qquad g_{\textrm{call}}\left( x\right) = \left( \textrm{e}^x-\mathscr {K}\right) ^{+}, $$\end{document}

and discretising over the partition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi =\{0=t_0, t_1, \dots t_N=T\}$$\end{document} for some  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N\in \mathbb {N}$$\end{document} . The precise discretisation schemes of the individual processes are given in their corresponding sections below. We remark, however, that the approximated option price for a given Monte-Carlo sample can become (slightly) negative by construction so we add an absorption feature for both models:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y^\pi _{t_i} :=\max \left\{ 0,{\widetilde{Y}}^\pi _{t_i}\right\} , \qquad {\text {for }} i\in \{0, \dots , N-1\}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\big \{{\widetilde{Y}}^\pi _{t_i}\big \}_{i=0}^N$$\end{document} denotes the approximation obtained through the RWNN scheme.

Remark 6.1

This is a well-studied problem and is especially prevalent in the simulation of square-root diffusions. We acknowledge that the absorption scheme possibly creates additional bias (see [49] for the case of the Heston model), however, a theoretical study in the case of the PDE-RWWN scheme is out of the scope of this paper.

The reservoir used as a random basis of RWNNs here is the classical linear reservoir from Definition 2.2. For numerical purposes, we introduce a so-called connectivity parameter, a measure of how interconnected the neurons in a network are: the higher the connectivity, the more inter-dependence between the neurons (see [17] for effects of connectivity in different reservoir topologies). In practice, however, too high a connectivity can lead to overfitting and poor generalisation. Recall that our reservoir is given by

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \Phi _K: \mathbb {R}^d \rightarrow \mathbb {R}^K, \qquad x\mapsto \Phi _K(x):=\boldsymbol{\varrho }(\textrm{A}x+\textrm{b}), $$\end{document}

where only \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{A}\in \mathbb {R}^{K\times d}$$\end{document} is affected by the connectivity parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c\in (0,1]$$\end{document} . Mathematically, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{A}_{ij} = {\tilde{\textrm{A}}}_{ij}1\hspace{-2.1mm}{1}_{\{Z_{ij} < c\}}$$\end{document} , where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_{ij} \overset{\textrm{iid}}{\sim } \mathcal {U}_{[0,1]}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\tilde{\textrm{A}}}_{ij}$$\end{document} is the original matrix not impacted by the connectivity parameter. A value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=1$$\end{document} means that  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{A}$$\end{document} is dense and fully determined by sampled weights. We find that the choice \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c\approx 0.5$$\end{document} results in superior performance.

In all our experiments, the reservoir weights are sampled uniformly over \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[-0.5, 0.5]$$\end{document} (i.e. with the choice \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R=0.5$$\end{document} in Algorithms 1 and 2). We experimented with sampling over alternative distributions and/or intervals, yet discovered that the scheme remains robust to the choice of support, provided it aligns with the magnitude of the expected output. All experiments below were run on a standard laptop with an AMD Ryzen 9 5900HX processor without any use of GPU, which would most certainly speed up the algorithms further. The code for both models is available at ZuricZ/RWNN_PDE_solver.

Example: Black-Scholes

The Black-Scholes model [12] is ubiquitous in mathematical finance, allowing for closed-form pricing and hedging of many financial contracts. Despite its well-known limitations, it remains a reference and is the first model to check before exploring more sophisticated ones. Since it offers an analytical pricing formula as a benchmark for numerical results, it will serve as a proof of concept for our numerical scheme. Under the pricing measure  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {Q}$$\end{document} , the underlying assets \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{S}= (S^1,\ldots , S^d)$$\end{document} satisfy

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textrm{d}S^j_t = S^j_t \left( r \textrm{d}t + \sigma _j \textrm{d}W^j_t\right) , \quad {\text {for }} t\in [0,T],\; j\in \{1,\dots ,d\}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{W^j_t\}_{t\in [0,T]}$$\end{document} are standard Brownian motions such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\langle W^i, W^j \rangle = \rho _{i,j}\textrm{d}t$$\end{document} , with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _{i,j}\in [-1,1]$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r\ge 0$$\end{document} is the risk-free rate and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _j>0$$\end{document} is the volatility coefficient. The corresponding d-dimensional option pricing PDE is then given by

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{\partial u(t,\boldsymbol{S})}{\partial t}&+\sum _{j=1}^d r S^j \frac{\partial u(t,\boldsymbol{S})}{\partial S^j} + \sum _{j=1}^d \frac{(\sigma _i S^j)^2}{2} \frac{\partial ^2 u(t,\boldsymbol{S})}{(\partial S^j)^2} \\&+ \sum _{j=1}^{d-1} \sum _{k=j+1}^d \rho _{j, k} \sigma _j \sigma _k S^j S^k \frac{\partial ^2 u(t,\boldsymbol{S})}{\partial S^j \partial S^k} = r u(t,\boldsymbol{S}), \end{aligned}$$\end{document}

for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\in [0,T)$$\end{document} with terminal condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$u(T,\boldsymbol{S}_T) = g(S^1_T,\dots , S^d_T)$$\end{document} . To use Algorithm 1, the process  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{S}$$\end{document} has to be discretised, for example with an Euler-Maruyama scheme, for each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j=1,\ldots , d$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \{0,1,\dots ,N-1\}$$\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{array}{ll} X^{\pi , j}_{t_{i+1}} & = X^{\pi , j}_{t_{i}} + \left( r-\frac{\sigma _j^2}{2}\right) \delta _{i}+ \sigma _j\Delta ^{W^j}_{i}, \\ S^{\pi , j}_{t_{i+1}} & = \exp \left\{ X^{\pi , j}_{t_{i+1}}\right\} , \end{array} \right. \end{aligned}$$\end{document}

with initial value \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X^{\pi , j}_0=\log \big (S^{\pi , j}_0\big )$$\end{document} . If not stated otherwise, we let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathscr {K}=S_0=1$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r=0.01$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T=1$$\end{document} , and run the scheme with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=21$$\end{document} discretisation steps and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{\textrm{MC}}=50,000$$\end{document} Monte-Carlo samples. The reservoir has \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K\in \{10, 100, 1000\}$$\end{document} hidden nodes, in Sections 6.1.4 and 6.1.2 the connectivity parameter is set to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=0.5$$\end{document} .

Convergence rate

We empirically analyse the error rate in terms of the number of hidden nodes K obtained in Corollary 5.5. To isolate the dependence on the number of nodes, we fix the discretisation grid and the number of MC samples. We then consider a single ATM vanilla Call, fix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=1$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma =0.1$$\end{document} and vary

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ K \in \left\{ \lfloor 10^{1 + \frac{2(i - 1)}{9}} \rfloor : i \in 1, \dots , 10 \right\} , $$\end{document}

over a set of 10 logarithmically spaced points between 10 and 1000. Due to our vectorised implementation of the algorithm, the reservoir basis tensor cannot fit into the random-access memory of a standard laptop for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K \ge 10000$$\end{document} . The results in Figure 1 are compared to the theoretical price only computed using the Black-Scholes pricing formula. The absorption scheme (6.1) is applied.Fig. 1. Empirical convergence of the MSE from Corollary 5.5 under Black-Scholes in terms of the number of hidden nodes (for a fixed grid and number of MC samples). Error bars mark 0.1 and 0.9 quantiles of 20 separate runs of the algorithm. The slope coefficient of the dashed line is obtained through regression of the means of individual runs, while the solid line represents 1/K convergence and is shown as a reference

ATM Call option

As a proof of concept we first test Algorithm 1 with Call options written on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d\in \mathbb {N}$$\end{document} independent assets, i.e. with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _{j,k}=0$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\ne k$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V_T = \big (g_{\textrm{call}}(S^j_T)\big )_{j\in \{1,\dots ,d\}}$$\end{document} . This is only done so that the results can be directly compared to the theoretical price computed using the Black-Scholes pricing formula and is, in effect, the same as pricing d options on d independent assets, each with their own volatility parameter  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma $$\end{document} . All the listed results in this section are for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=100$$\end{document} hidden nodes. In Table 1, results and relative errors are shown for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=5$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\sigma }:=(\sigma _1,\dots ,\sigma _d)$$\end{document} uniformly spaced over [0.05, 0.4]. Next, the effects of the absorption scheme (6.1) are investigated. Curiously, the absorption performs noticeably worse compared to the basic scheme, where one does not adjust for negative paths. This leads us to believe that absorption adds a substantial bias, similar to the Heston case (see Remark 6.1). Therefore, such a scheme should only be used for purposes, when positivity of the option price paths is strictly necessary (e.g. when hedging). Finally, in Table 2, total MSE and computational times are given for different dimensions. The computational times for different dimensions are then plotted in Figure 2. It is important to note that our results do not allow us to make definitive claims about the computational times of the PDE-RWNN scheme across different dimensions. This was not the goal of our experiments, and further detailed theoretical study and experiments would be necessary to draw more definitive conclusions regarding the efficiency of the scheme in various dimensions.Table 1A single run for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=5$$\end{document} independent underlyings, where European Calls are compared to the price obtained through PDE-RWNN (with and without absorption) and the Monte Carlo methods along each dimension. Below, the relative errors of both methods are given. The MC method was run using the same paths as in the PDE-RWNNPrice \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\sigma }$$\end{document} TruePDE (w/ abs)PDE (w/o abs)MC0.050.025216400.029602560.025311310.025747310.10.044852360.055231140.044676870.045475650.150.064594830.077199490.064776050.065207830.20.084333190.103078680.084439570.084849610.250.104035390.126608710.104123930.10513928Rel. Error \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\sigma }$$\end{document} PDE(w/ abs)PDE (w/o abs)MC0.051.74e-01-3.76e-03-2.11e-020.12.31e-01 3.91e-03-1.39e-020.151.95e-01-2.81e-03-9.49e-030.22.22e-01-1.26e-03-6.12e-030.252.17e-01-8.51e-04-1.06e-02Table 2Total MSE of the option price calculated across all d assets and CPU training times for varying dimension d, where  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\sigma }$$\end{document} uniformly spaced over [0.05, 0.4]dTotal MSE (with abs)CPU Time (seconds)53.482e-810.5105.417e-816.0254.901e-834.5501.653e-765.01002.534e-7135.0

Fig. 2. Computational time vs number of dimensions, as in Table 2

Computational time

A key advantage of RWNNs is their fast training procedure, which in essence relies on solving a linear regression problem. We now look at (2.3) to assess its computational complexity. First, computing the sum of outer products, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum _{j=1}^n Y_j X_j^\top $$\end{document} , where each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_j \in \mathbb {R}^d$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_j \in \mathbb {R}^k$$\end{document} , requires forming n matrices of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d \times k$$\end{document} , resulting in a total computational cost of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(ndk)$$\end{document} . Similarly, for the sum \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum _{j=1}^n X_j X_j^\top $$\end{document} , each outer product produces a matrix of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \times k$$\end{document} , yielding a cost of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(nk^2)$$\end{document} . Multiplying the resulting matrices requires \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(dk^2)$$\end{document} operations, and inversion of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k \times k$$\end{document} matrix incurs a cost of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(k^3)$$\end{document} . The overall computational complexity is therefore

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathcal {O}(ndk + nk^2 + k^3), $$\end{document}

where, for large n, the sum over n outer products is typically the dominant term. The higher-order terms \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(nk^2)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(k^3)$$\end{document} only become significant when k is no longer negligible in comparison to n and d. The complexity estimate is consistent with the empirical results presented in Figure 2, which demonstrate a linear relationship between the number of dimensions d and observed CPU time. In effect, the dominant \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(ndk)$$\end{document} term shows the scheme mitigates the curse of dimensionality, allowing high-dimensional problems to be tackled efficiently within this framework.

Basket option

We consider an equally weighted basket Call option with a payoff

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ g_{basket }(\boldsymbol{S}_T):=\left( \frac{1}{d}\sum _{j=1}^d S^j_T - {\mathscr {K}}\right) ^+, $$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathscr {K}}>0$$\end{document} denotes the strike price. For simplicity, we consider \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=5$$\end{document} and an ATM option with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathscr {K}}:=\frac{1}{d}\sum _{j=1}^d S^j_0$$\end{document} and set all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_0^j=1$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\in \{1,\dots ,5\}$$\end{document} . The volatilities \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _j$$\end{document} are uniformly spaced between [0.05, 0.25] and the correlation matrix is randomly chosen as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \boldsymbol{\rho }:=\begin{bmatrix} 1 & 0.84 & -0.51 & -0.70 & 0.15 \\ 0.84 & 1 & -0.66 & -0.85 & 0.41 \\ -0.51 & -0.66 & 1 & 0.55 & -0.82 \\ -0.70 & -0.85 & 0.55 & 1 & -0.51 \\ 0.15 & 0.41 & -0.82 & -0.51 & 1 \end{bmatrix}, $$\end{document}

so that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma :=\operatorname {diag}(\boldsymbol{\sigma })\boldsymbol{\rho }\operatorname {diag}(\boldsymbol{\sigma })$$\end{document} . Since the distribution of a sum of Lognormal is not known explicitly, no closed-form expression is available for the option price. Hence, the reference price is computed using Monte-Carlo with 100 time steps and 400, 000 samples. In Table 3, we compare our scheme with a classical MC estimator in terms of relative error for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K = 100$$\end{document} hidden nodes.Table 3. Comparison of prices, relative errors and CPU time of the Monte-Carlo estimator, PDE-RWNN scheme with and without absorption (using same sampled MC paths and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K = 100$$\end{document} ) and the reference price computed with 100 time steps and 400,000 samplesReferencePDE (with abs)PDE (without abs)MCPrice0.016240.018220.016130.01625Rel. error--1.22e-01-6.71e-03-6.50e-04Time (seconds)12.89.79.80.3

Example: Rough Bergomi

The rough Bergomi model belongs to the recently developed class of rough stochastic volatility models, first proposed in [4, 27, 34], where the instantaneous variance is driven by a fractional Brownian motion (or more generally a continuous Gaussian process) with Hurst parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H < \frac{1}{2}$$\end{document} . As highlighted in many papers, they are able to capture many features of (Equities, Commodities,...) data, and clearly seem to outperform most classical models, with fewer parameters. Precise examples with real data can be found in [4] for SPX options, in [28, 38] for joint SPX-VIX options and in [9, 27] for estimation on historical time series, the latter being the state-of-the-art statistical analysis under the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {P}$$\end{document} -measure. We consider here the price dynamics under  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {Q}$$\end{document} with constant initial forward variance curve  \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _0(t)>0$$\end{document} for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\in [0,T]$$\end{document} :

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{array}{r@{\;}l} \displaystyle \frac{\textrm{d}S_t}{S_t} & = r \textrm{d}t + \sqrt{V_t} \textrm{d}\left( \rho \textrm{d}W^1_t + \sqrt{1-\rho ^2} W^2_t\right) , \\ V_t & = \displaystyle \xi _0(t)\mathcal {E}\left( \eta \sqrt{2H} \int _0^t (t-u)^{H-\frac{1}{2}} \textrm{d}W_u^1 \right) , \end{array} \right. \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta >0$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho \in (-1,1)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H\in (0,1)$$\end{document} . The corresponding BSPDE reads

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} -\textrm{d}u(t,x) =&\left[ \frac{V_{t}}{2} \partial _x^{2} u(t, x)+\rho \sqrt{V_{t}} \partial _x \psi (t,x)-\frac{V_{t}}{2} \partial _x u(t,x)-r u(t,x)\right] \textrm{d}t \\&-\psi (t,x) \textrm{d}W^1_{t}, \end{aligned}$$\end{document}

with terminal condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$u(T, x) = g_{\textrm{call}}\left( \textrm{e}^{x+r T}\right) $$\end{document} . While the existence of the solution was only proven in the distributional sense [6], we nevertheless apply our RWNN scheme. To test Algorithm 2, both the price and the volatility processes are discretised according to the Hybrid scheme developed in [10, 51]. We set the rBergomi parameters as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(H, \eta , \rho , r, T, S_0) = (0.3, 1.9, -0.7, 0.01, 1, 1)$$\end{document} and choose the forward variance curve to be flat with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _0(\cdot ) = 0.235^2$$\end{document} . Again, we are pricing an ATM vanilla Call option with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathscr {K}}=S_0=1$$\end{document} . The number of discretisation steps is again \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=21$$\end{document} , the number of Monte-Carlo samples \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{\textrm{MC}}=50,000$$\end{document} and the reservoir has \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K\in \{10, 100, 1000\}$$\end{document} nodes with connectivity \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=0.5$$\end{document} in Section 6.2.2.

Convergence rate

As in Section 6.1.1, we conduct an empirical analysis of the convergence error from Corollary 5.5 for the same ATM Call. To isolate the dependence on the number of nodes we fix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=1$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{\textrm{MC}}=50,000$$\end{document} and vary

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ K \in \left\{ \lfloor 10^{1 + \frac{2(i - 1)}{9}} \rfloor : i \in 1, \dots , 10 \right\} , $$\end{document}

logarithmically spaced points between 10 and 1000. The reference price is computed by Monte-Carlo with 100 time steps and 800, 000 samples. The absorption scheme has been applied and the results are displayed in Figure 3. In this section, the same random seed was used as in Section 6.1.1, to ensure consistent results across different simulations.Fig. 3. Empirical convergence of MSE under rBergomi in terms of the number of hidden nodes. Error bars mark 0.1 and 0.9 quantiles of 20 separate runs of the algorithm. The slope coefficient of the dashed line is obtained through regression of the means of individual runs, while the solid line represents 1/K convergence and is shown as a reference

ATM Call option

We now evaluate the performance of our PDE-RWNN method for option pricing in the rough Bergomi model using the market parameters listed above and compare the results to those obtained with the MC method over the same sample paths. We also investigate the effect of the absorption scheme (Table 4) and find that, interestingly, despite keeping the paths positive, the absorption scheme adds noticeable bias. Nevertheless, the relative error of the proposed scheme with absorption is comparable to the results using regular artificial networks found in the literature [6, Table 1], yet, our scheme learns much faster with orders of magnitudes faster training times.Table 4. Prices, relative errors and CPU time of the Monte-Carlo estimator, PDE-RWNN scheme with absorption, PDE-RWNN scheme without absorption, both with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K = 100$$\end{document} and same sampled MC paths. Reference price computed with 100 time steps and 800,000 samplesReferencePDE (with abs)PDE (without abs)MCPrice0.079930.0819240.079730.080310Rel. error-24.9e-032.54e-03-4.73e-03Time (seconds)10.17.47.50.4