Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis

TL;DR

This paper introduces deep learning algorithms for high-dimensional stochastic control problems, providing theoretical convergence analysis and demonstrating their effectiveness through numerical experiments.

Contribution

It develops neural network-based algorithms for stochastic control, with rigorous convergence analysis and error estimates, advancing the application of deep learning in control theory.

Findings

Algorithms converge with rates depending on neural network approximation errors.

Theoretical analysis confirms consistency of control and value function estimates.

Numerical results show promising performance in high-dimensional problems.

Abstract

This paper develops algorithms for high-dimensional stochastic control problems based on deep learning and dynamic programming. Unlike classical approximate dynamic programming approaches, we first approximate the optimal policy by means of neural networks in the spirit of deep reinforcement learning, and then the value function by Monte Carlo regression. This is achieved in the dynamic programming recursion by performance or hybrid iteration, and regress now methods from numerical probabilities. We provide a theoretical justification of these algorithms. Consistency and rate of convergence for the control and value function estimates are analyzed and expressed in terms of the universal approximation error of the neural networks, and of the statistical error when estimating network function, leaving aside the optimization error. Numerical results on various applications are presented in a companion paper (arxiv.org/abs/1812.05916) and illustrate the performance of the proposed algorithms.

Equations

$$X_{n+1}^{\alpha }$$

$$V_0(x_0)$$

$$\left\{\begin{array}[]{rcl}Q_{n}(x,a)&=&f(x,a)+P^{a}V_{n+1}(x),\;\;\;x\in{\cal X},\;a\in\mathbb{A},\\ V_{n}(x)&=&\displaystyle\inf_{a\in\mathbb{A}}Q_{n}(x,a),\end{array}\right.$$

$$X_{n+1}^{\ast }$$

$$\displaystyle\inf_{\theta}\mathbb{E}\big{[}L(Z;\theta)\big{]},$$

$${\hat{\beta }}_n$$

$${\hat{V}}_n^M(x)={\mathbb{E}}_M\left[\sum _{k=n}^{N-1}f({\hat{X}}_k^{n,x},{\hat{a}}_k^M({\hat{X}}_k^{n,x}))+g({\hat{X}}_N^{n,x})\right],$$

$${\hat{\beta }}_n$$

$${\hat{V}}_n^M$$

$$P^a(x,d{x}^{\prime })$$

$$|r(x,a;x^{\prime })|$$

$$|r(x_1,a_1;x^{\prime })-r(x_2,a_2;x^{\prime })|$$

$$|f(x,a)|$$

$$|f(x_1,a_1)-f(x_2,a_2)|$$

$$|g(x_1)-g(x_2)|$$

$$\left|F(x,a,e)-F(x^{\prime },a^{\prime },e)\right|\le C(e)\left(|x-x^{\prime }|+|a-a^{\prime }|\right).$$

$${\rho }_M\le [b{]}_L+d[\sigma {]}_L\sqrt{2\log (2dM)},$$

$$e^{t\mathbb{E}\left[Z\right]}$$

$$\displaystyle\mathbb{E}\big{[}Z\big{]}\leq\frac{\log(2M^{\prime})}{t}+\frac{t}{2}.$$

$$\displaystyle\mathbb{E}\big{[}Z\big{]}\leq\sqrt{2\log(2M^{\prime})}.$$

$$\mathbb{E}\left[\underset{1\le m\le M}{\sup }C({\epsilon }^m)\right]\le [b{]}_L+d[\sigma {]}_L\mathbb{E}\left[\underset{1\le m\le dM}{\sup }C({ϵ}_1^m)\right],$$

$$\left\{\begin{array}[]{ccl}[V_{N}]_{L}&=&[g]_{L}\\ \left[V_{n}\right]_{L}&\leq&\min\left([f]_{L}+\lVert V_{n}\rVert_{\infty}[r]_{L}\;,\;\rho_{1}\frac{1-\rho_{1}^{N-n}}{1-\rho_{1}}+\rho_{1}^{N-n}[g]_{L}\right),\text{ for }n=0,\ldots,N-1,\end{array}\right.$$

$${}^{\eta }{\mathcal{A}}_K^{\gamma }$$

$$\begin{array}{r}{K}_M,{\gamma }_M,{\eta }_M\underset{M\to \infty }{\to }\infty \text{and}{\rho }_M^{N-1}{\gamma }_M^{N-1}{\eta }_M^{N-2}\sqrt{\frac{\log (M)}{M}}\underset{M\to \infty }{\to }0.\end{array}$$

$$\begin{array}[]{ccc}\hat{a}_{n}^{M}&\in&\underset{A\in\mathcal{A}_{M}}{\mathrm{argmin\,}}\frac{1}{M}\sum_{m=1}^{M}\left[f(X_{n}^{(m)},A(X_{n}^{(m)}))+\hat{Y}^{(m),A}_{n+1}\right]\end{array}$$

$${\hat{Y}}_{n+1}^{(m),A}=\sum _{k=n+1}^{N-1}f\left(X_k^{(m),A},{\hat{a}}_k^M\left(X_k^{(m),A}\right)\right)+g\left(X_N^{(m),A}\right),$$

$$\left\{\begin{array}[]{ccl}X^{(m),A}_{n+1}&=&F\Big{(}X^{m}_{n},A\big{(}X^{m}_{n}\big{)},\varepsilon_{n+1}^{m}\Big{)}\\ X^{(m),A}_{k}&=&F\Big{(}X^{(m),A}_{k-1},\hat{a}_{k}^{M}\big{(}X^{(m),A}_{k-1}\big{)},\varepsilon_{k}^{m}\Big{)},\;\;\;\mbox{ for }k=n+2,\ldots,N.\end{array}\right.$$

$$\begin{split}\mathbb{E}\big{[}\hat{V}_{n}^{M}(X_{n})-V_{n}(X_{n})\big{]}&=\mathcal{O}\Bigg{(}\frac{\rho_{M}^{N-n-1}\gamma_{M}^{N-n-1}\eta_{M}^{N-n-2}}{\sqrt{M}}\\ &\hskip 25.6073pt+\displaystyle\sup_{n\leq k\leq N-1}\inf_{A\in\mathcal{A}_{M}}\mathbb{E}\left[|A(X_{k})-a^{\mathrm{opt}}_{k}(X_{k})|\right]\Bigg{)},\end{split}$$

$$\begin{split}\mathbb{E}_{M}\big{[}\hat{V}_{n}^{M}(X_{n})-V_{n}(X_{n})\big{]}&={\cal O}_{\mathbb{P}}\Bigg{(}\rho_{M}^{N-n-1}\gamma_{M}^{N-n-1}\eta_{M}^{N-n-2}\sqrt{\frac{\log(M)}{M}}\\ &\hskip 34.14322pt+\sup_{n\leq k\leq N-1}\inf_{A\in\mathcal{A}_{M}}\mathbb{E}\left[|A(X_{k})-a^{\mathrm{opt}}_{k}(X_{k})|\right]\Bigg{)},\end{split}$$

$${\sup_{n\leq k\leq N}\inf_{A\in\mathcal{A}_{M}}\mathbb{E}\big{[}|A(X_{k})-a^{\mathrm{opt}}_{k}(X_{k})|\big{]}\xrightarrow[M\to+\infty]{}0}\ .$$

$$\sup_{n\leq k\leq N-1}\inf_{A\in\mathcal{A}_{M}}\mathbb{E}\big{[}|A(X_{k})-a^{\mathrm{opt}}_{k}(X_{k})|\big{]}\xrightarrow[M\to+\infty]{}0.$$

$$\displaystyle\sup_{n\leq k\leq N-1}\inf_{A\in\mathcal{A}_{M}}\;\mathbb{E}\big{[}|A(X_{k})-a^{\mathrm{opt}}_{k}(X_{k})|\big{]}$$

Full text

{AMS}

65C05, 90C39, 93E35

1 Introduction

A large class of dynamic decision-making problems under uncertainty can be mathematically modeled as discrete-time stochastic optimal control problems in finite horizon. This paper is devoted to the analysis of novel probabilistic numerical algorithms based on neural networks for solving such problems. Let us consider the following discrete-time stochastic control problem over a finite horizon $N$ $\in$ $\mathbb{N}\setminus\{0\}$. The dynamics of the controlled state process $X^{\alpha}$ $=$ $(X^{\alpha}_{n})_{n}$ valued in ${\cal X}$ $\subset$ $\mathbb{R}^{d}$ is given by

[TABLE]

where $(\varepsilon_{n})_{n}$ is a sequence of i.i.d. random variables valued in some Borel space $(E,{\cal B}(E))$, and defined on some probability space $(\Omega,{\cal F},\mathbb{P})$ equipped with the filtration $\mathbb{F}$ $=$ $({\cal F}_{n})_{n}$ generated by the noise $(\varepsilon_{n})_{n}$ (${\cal F}_{0}$ is the trivial $\sigma$-algebra), the control $\alpha$ $=$ $(\alpha_{n})_{n}$ is an $\mathbb{F}$-adapted process valued in $\mathbb{A}$ $\subset$ $\mathbb{R}^{q}$, and $F$ is a measurable function from $\mathbb{R}^{d}\times\mathbb{R}^{q}\times E$ into $\mathbb{R}^{d}$.

Given a running cost function $f$ defined on $\mathbb{R}^{d}\times\mathbb{R}^{q}$, a terminal cost function $g$ defined on $\mathbb{R}^{d}$, the cost functional associated with a control process $\alpha$ is $J(\alpha)=\mathbb{E}\left[\sum_{n=0}^{N-1}f(X_{n}^{\alpha},\alpha_{n})+g(X_{N}^{\alpha})\right].$ The set ${\cal C}$ of admissible control is the set of control processes $\alpha$ satisfying some integrability conditions ensuring that the cost functional $J(\alpha)$ is well-defined and finite. The control problem, also called Markov decision process (MDP), is formulated as

[TABLE]

and the goal is to find an optimal control $\alpha^{*}$ $\in$ ${\cal C}$, i.e., attaining the optimal value: $V_{0}(x_{0})$ $=$ $J(\alpha^{*})$. Notice that problem (1)-(2) may also be viewed as the time discretization of a continuous time stochastic control problem, in which case $F$ is typically the Euler scheme for a controlled diffusion process, and $V_{0}$ is the discrete-time approximation of a fully nonlinear Hamilton-Jacobi-Bellman equation.

Problem (2) is tackled by the dynamic programming approach, and we introduce the standard notations for MDP: denote by $\{P^{a}(x,dx^{\prime})$, $a$ $\in$ $\mathbb{A}$, $x$ $\in$ ${\cal X}\}$, the family of transition probabilities associated with the controlled (homogenous) Markov chain (1), given by $P^{a}(x,dx^{\prime}):=\mathbb{P}\left[F(x,a,\varepsilon_{1})\in dx^{\prime}\right]$ and for any measurable function $\varphi$ on ${\cal X}$: P^{a}\varphi(x)=\int\varphi(x^{\prime})P^{a}(x,dx^{\prime})=\mathbb{E}\left[\varphi\big{(}F(x,a,\varepsilon_{1})\big{)}\right]. With these notations, we have for any measurable function $\varphi$ on ${\cal X}$, for any $\alpha$ $\in$ ${\cal C}$, $\mathbb{E}[\varphi(X_{n+1}^{\alpha})|{\cal F}_{n}]=P^{\alpha_{n}}\varphi(X_{n}^{\alpha}),\;\;\;\forall\;n\in\mathbb{N}.$ The optimal value $V_{0}(x_{0})$ is then determined in backward induction starting from the terminal condition $V_{N}(x)=g(x),\;x\in{\cal X},$ and by the dynamic programming (DP) formula, for $n$ $=$ $N-1,\ldots,0$:

[TABLE]

The function $Q_{n}$ is called optimal state-action value function, and $V_{n}$ is the (optimal) value function. Moreover, when the infimum is attained in the DP formula at any time $n$ by $a_{n}^{*}(x)$, we get an optimal control in feedback form given by: $\alpha^{*}$ $=$ $(a_{n}^{*}(X_{n}^{*}))_{n}$ where $X^{*}$ $=$ $X^{\alpha^{*}}$ is the Markov process defined by

[TABLE]

The implementation of the DP formula requires the knowledge and explicit computation of the transition probabilities $P^{a}(x,dx^{\prime})$. In situations when they are unknown, this leads to the problematic of reinforcement learning for computing the optimal control and value function by relying on simulations of the environment. The challenging tasks from a numerical point of view are then twofold:

Transition probability operator. Calculations for any $x$ $\in$ ${\cal X}$, and any $a$ $\in$ $\mathbb{A}$, of $P^{a}V_{n+1}(x)$, for $n$ $=$ $0,\ldots,N-1$. This is a computational challenge in high dimension $d$ for the state space with the “curse of dimensionality” due to the explosion of grid points in deterministic methods.

• 2.

Optimal control. Computation of the infimum in $a$ $\in$ $\mathbb{A}$ of $f(x,a)+P^{a}V_{n+1}(x)$ for fixed $x$ and $n$, and of $\hat{a}_{n}(x)$ attaining the minimum if it exists. This is also a computational challenge especially in high dimension $q$ for the control space.

The classical probabilistic numerical methods based on DP for solving the MDP are sometimes called approximate dynamic programming methods, see e.g. [6], [23], and consist basically of the two following steps:

• (i)

Approximate the $Q_{n}$. This can be performed by regression Monte Carlo (RMC) techniques or quantization. RMC is typically done by least squares linear regression on a set of basis function following the popular approach by Longstaff and Schwartz [21] initiated for the Bermudan option problem, where the suitable choice of basis functions might be delicate. Conditional expectation can also be approximated by regression on neural networks as in [17] for the American option problem and appears as a promising and efficient alternative in high dimension.

• (ii)

Control search. Once we get an approximation $(x,a)$ $\mapsto$ $\hat{Q}_{n}(x,a)$ of the $Q_{n}$ value function, the optimal control $\hat{a}_{n}(x)$ which achieves the minimum over $a$ $\in$ $\mathbb{A}$ of $Q_{n}(x,a)$ can be obtained either by an exhaustive search when $\mathbb{A}$ is discrete (with relatively small cardinality), or by a (deterministic) gradient-based algorithm for continuous control space (with relatively small dimension).

Recently, numerical methods by direct approximation, without DP, have been developed and made implementable thanks to the power of computers: the basic idea is to focus directly on the control approximation by considering feedback control (policy) in a parametric form: $a_{n}(x)=A(x;\theta_{n}),\;\;\;n=0,\ldots,N-1,$ for some given function $A(.,\theta_{n})$ with parameters $\theta$ $=$ $(\theta_{0},\ldots,\theta_{N-1})$ $\in$ $\mathbb{R}^{q\times N}$, and minimize over $\theta$ the parametric functional $\tilde{J}(\theta)=\mathbb{E}\left[\sum_{n=0}^{N-1}f(X_{n}^{A},A(x;\theta_{n}))+g(X_{N}^{A})\right],$ where $(X_{n}^{A})_{n}$ denotes the controlled process with feedback control $(A(.,\theta_{n}))_{n}$. This approach was first adopted in [19], which used the EM algorithm for optimizing over the parameter $\theta$, and was further investigated in [11], [8], [13], which considered deep neural networks (DNN) for the parametric feedback control, and stochastic gradient descent methods (SGD) for computing the optimal parameter $\theta$. The theoretical foundation of these DNN algorithms has been recently investigated in [12]. Let us mention that DNN approximation in stochastic control has already been explored in the context of reinforcement learning (RL) (see [6] and [24]), and is called deep reinforcement learning in the artificial intelligence community [22] (see also [20] for a recent survey) but usually for infinite horizon (stationary) control problems.

In this paper, we combine different ideas from the mathematics (numerical probability) and the computer science (reinforcement learning) communities to propose and compare several algorithms based on dynamic programming (DP), and deep neural networks (DNN) for the approximation/learning of (i) the optimal policy, and then of (ii) the value function. Notice that this differs from the classical approach in DP recalled above, where we first approximate the $Q$-optimal state/control value function, and then approximate the optimal control. Our learning of the optimal policy is achieved in the spirit of [11] by DNN, but sequentially in time through DP instead of a global learning over the whole period $0,\ldots,N-1$. Once we get an approximation of the optimal policy, we approximate the value function by Monte Carlo (MC) regression based on simulations of the forward process with the approximated optimal control. In particular, we avoid the issue of a priori endogenous simulation of the controlled process in the classical $Q$-approach. The MC regressions for the approximation of the optimal policy and/or value function, rely on performance iteration (PI) or hybrid iteration (HI). Numerical results on several applications are devoted to a companion paper [3]. The theoretical contribution of the current paper is to provide a detailed convergence analysis of our two proposed algorithms: Theorem 4.6 for the *NNContPI * algorithm based on control learning by performance iteration with DNN, and Theorem 4.13 for the Hybrid-Now algorithm based on control learning by DNN and then value function learning by the regress-now method. We rely mainly on arguments from statistical learning and non parametric regression as developed notably in the book [10] to give estimates of approximated control and value function in terms of the universal approximation error of the neural networks, and of the statistical error in the estimation of network functions. In particular, the estimation of the optimal control using the control that minimizes the empirical loss is studied based on the uniform law of large numbers (see [10]); and the estimation of the value function in Hybrid-Now relies on Lemma H.1, proved in [17], which bounds the difference between the conditional expectation and the best approximation of the empirical conditional expectation by neural networks. In practical implementation, the computation of the approximate optimal policy in the minimization of the empirical loss function is performed by a stochastic gradient descent algorithm, which leads to the so-called optimization error, see e.g. [5], [15]. We do not address in the current paper this type of error in machine learning algorithms, and we refer to the recent paper [4] for a full error analysis of deep neural networks training.

The paper is organized as follows. We recall in Section 2 some basic results about deep neural networks (DNN) and stochastic optimization gradient descent methods. Section 3 is devoted to the description of our two algorithms. We analyze in Section 4 the convergence of the algorithms. Finally the Appendices collect some Lemmas used in the proof of the convergence results.

2 Preliminaries on DNN and SGD

2.1 Neural network approximations

Deep Neural networks (DNN) aim to approximate (complex nonlinear) functions defined on finite-dimensional space, and in contrast with the usual additive approximation theory built via basis functions, like polynomial, they rely on composition of layers of simple functions. The relevance of neural networks comes from the universal approximation theorem and the Kolmogorov-Arnold representation theorem (see [18], [7] or [14]), and this has been shown to be successful in numerous practical applications.

We consider here a feedforward artificial network (also called multilayer perceptron) for the approximation of the optimal policy (valued in $\mathbb{A}$ $\subset$ $\mathbb{R}^{q}$) and the value function (valued in $\mathbb{R}$), both defined on the state space ${\cal X}$ $\subset$ $\mathbb{R}^{d}$. The architecture is represented by functions $x\;\in\;{\cal X}\longmapsto\Phi(z;\theta)\;\in\mathbb{R}^{o},$ with $o$ $=$ $q$ or $1$ in our context, and where $\theta$ $\in$ $\Theta$ $\subset$ $\mathbb{R}^{p}$ are the weights (or parameters) of the neural networks.

2.2 Stochastic optimization in DNN

Approximation by means of DNN requires a stochastic optimization with respect to a set of parameters, which can be written in a generic form as

[TABLE]

where $Z$ is a random variable from which the training samples $Z^{(m)}$, $m$ $=$ $1,\ldots,M$ are drawn, and $L$ is a loss function involving DNN with parameters $\theta$ $\in$ $\mathbb{R}^{p}$, and typically differentiable w.r.t. $\theta$ with known gradient $D_{\theta}L(Z^{(m)})$.

Several algorithms based on stochastic gradient-descent are often used in practice for the search of infimum in (4); see e.g. basic SGD, AdaGrad, RMSProp, or Adam.

In the theoretical study of the algorithms proposed in this paper, we did not address the convergence of the gradient-descent algorithms, and rather assumed that the agent successfully solved the minimization problem (4), i.e. minimized the empirical loss $\sum_{m=1}^{M}L(Z^{(m)})$ using (non stochastic) gradient-descent, a stochastic gradient-descent method with many epochs or any other methods that allow overfitting. One can then rely on statistical learning results as in [10] and [17] that are specific to neural networks. Notice that overfitting should be avoided in practice, and so our framework remains theoretical.

3 Description of the algorithms

Let us introduce a set ${\cal A}$ of neural networks for approximating optimal policies, that is a set of parametric functions $x$ $\in$ ${\cal X}$ $\mapsto$ $A(x;\beta)$ $\in$ $\mathbb{A}$, with parameters $\beta$ $\in$ $\mathbb{R}^{l}$, and a set ${\cal V}$ of neural networks functions for approximating value functions, that is a set of parametric functions $x$ $\in$ ${\cal X}$ $\mapsto$ $\Phi(x;\theta)$ $\in$ $\mathbb{R}$, with parameters $\theta$ $\in$ $\mathbb{R}^{p}$.

We are also given at each time $n$ a probability measure $\mu_{n}$ on the state space ${\cal X}$, which we refer to as a training distribution. Some comments about the choice of the training measure are discussed in Section 3.3.

3.1 Control learning by performance iteration

This algorithm, referred in short as NNcontPI algorithm, is designed as follows:

For $n$ $=$ $N-1,\ldots,0$, keep track of the approximated optimal policies $\hat{a}_{k}$, $k$ $=$ $n+1,\ldots,N-1$, and approximate the optimal policy at time $n$ by $\hat{a}_{n}$ $=$ $A(.;\hat{\beta}_{n})$ with

[TABLE]

where $X_{n}$ $\sim$ $\mu_{n}$, $\hat{X}_{n+1}^{\beta}$ $=$ $F(X_{n},A(X_{n};\beta),\varepsilon_{n+1})$ $\sim$ $P^{A(X_{n};\beta)}(X_{n},dx^{\prime})$, and for $k$ $=$ $n+1,\ldots,N-1$, $\hat{X}_{k+1}^{\beta}$ $=$ $F(\hat{X}_{k}^{\beta},\hat{a}_{k}(\hat{X}_{k}^{\beta}),\varepsilon_{k+1})$ $\sim$ $P^{\hat{a}_{k}(\hat{X}_{k}^{\beta})}(\hat{X}_{k}^{\beta},dx^{\prime})$. Given estimate $\hat{a}_{k}^{M}$ of $\hat{a}_{k}$, $k$ $=$ $n+1,\ldots,N-1$, the approximated policy $\hat{a}_{n}$ is estimated by using a training sample $\left(X_{n}^{(m)},(\varepsilon_{k+1}^{(m)})_{k=n}^{k=N-1}\right)$, $m$ $=$ $1,\ldots,M$ of $\left(X_{n},(\varepsilon_{k+1})_{k=n}^{k=N-1}\right)$ for simulating $\left(X_{n},(\hat{X}_{k+1}^{\beta})_{k=n}^{k=N-1}\right)$, and optimizing $\beta$ of $A(.;\beta)$ $\in$ ${\cal A}$, by SGD (or its variants) as described in Section (2.2).

The estimated value function at time $n$ is then given as

[TABLE]

where $\mathbb{E}_{M}$ is the expectation conditioned on the training set used for computing $\left(\hat{a}_{k}^{M}\right)_{k}$; and $\left(\hat{X}_{k}^{n,x}\right)_{k=n}^{N}$ is driven by the estimated optimal controls. The dependence of the estimated value function $\hat{V}_{n}^{M}$ upon the training samples $X_{k}^{(m)}$, for $m$ $=$ $1,\ldots,M$, used at time $k$ $=$ $n,\ldots,N$, is emphasized through the exponent $M$ in the notations.

Remark 3.1

The NNcontPI algorithm can be viewed as a combination of the DNN algorithm designed in [11] and dynamic programming. In the algorithm presented in [11], which totally ignores the dynamic programming principle, one learns all the optimal controls $A(.;\beta_{n})$, $n$ $=$ $0,\ldots,N-1$ at the same time, by performing one unique stochastic gradient descent. This is efficient as all the parameters of all the NN are getting trained at the same time, using the same mini-batches. However, when the number of layers of the global neural network gathering all the $A(.;\beta_{n})$, for $n=0,\ldots,N-1$ is large (say $\sum_{n=0}^{N-1}\ell_{n}$ $\geq$ $100$, where $\ell_{n}$ is the number of layers in $A(.,\beta_{n})$), then one is likely to observe vanishing or exploding gradient problems that will affect the training of the weights and bias of the first layers of the global NN (see [9] for more details). * $\Box$*

Remark 3.2

The NNcontPI algorithm does not require value function iteration, but instead is based on performance iteration by keeping track of the estimated optimal policies computed in backward recursion. The value function is then computed in (6) as the gain functional associated with the estimated optimal policies $(\hat{a}_{k}^{M})_{k}$. Consequently, it provides usually a low bias estimate but induces possibly high variance estimate and large complexity, especially when $N$ is large. * $\Box$*

3.2 Control learning by hybrid iteration

Instead of keeping track of all the approximated optimal policies as in the NNcontPI algorithm, we use an approximation of the value function at time $n+1$ in order to compute the optimal policy at time $n$. The approximated value function is then updated at time $n$. This leads to the following algorithm:

Hybrid-Now algorithm

Initialization: $\hat{V}_{N}$ $=$ $g$

• 2.

For $n$ $=$ $N-1,\ldots,0$,

• (i)

Approximate the optimal policy at time $n$ by $\hat{a}_{n}$ $=$ $A(.;\hat{\beta}_{n})$ with

[TABLE]

where $X_{n}$ $\sim$ $\mu_{n}$, $\hat{X}_{n+1}^{A(.,\beta)}$ $=$ $F(X_{n},A(X_{n};\beta),\varepsilon_{n+1})$ $\sim$ $P^{A(X_{n};\beta)}(X_{n},dx^{\prime})$.

• (ii)

Updating: approximate the value function by NN:

[TABLE]

using samples $X_{n}^{(m)}$, $X_{n+1}^{\hat{a}_{n}^{M},(m)}$, $m$ $=$ $1,\ldots,M$ of $X_{n}$ $\sim$ $\mu_{n}$, and $X_{n+1}^{\hat{a}_{n}^{M},(m)}$ of $X_{n+1}^{\hat{a}_{n}^{M}}$

The approximated policy $\hat{a}_{n}$ is estimated by using a training sample $(X_{n}^{(m)},\varepsilon_{n+1}^{(m)})$, $m$ $=$ $1,\ldots,M$ of $(X_{n},\varepsilon_{n+1})$ to simulate $\left(X_{n},X_{n+1}^{A(.;\beta)}\right)$, and optimizing $A(.;\beta)$ $\in$ ${\cal A}$ over the parameters $\beta$ $\in$ $\mathbb{R}^{l}$ and the expectation in (7) by stochastic gradient descent methods as described in Section (2.2). We then get an estimate $\hat{a}_{n}^{M}$ $=$ $A\left(.;\hat{\beta}_{n}^{M}\right)$. The value function is represented by neural network and its optimal parameter is estimated by MC-regression in (8).

3.3 Training sets design

We discuss here the choice of the training measure $\mu_{n}$ used to generate the training sets on which will be computed the estimates. Two cases are considered in this section. The first one is a knowledge-based selection, relevant when the controller knows with a certain degree of confidence where the process has to be driven in order to optimize her cost functional. The second case, on the other hand, is when the controller has no idea where or how to drive the process to optimize the cost functional. Separating the construction of the training set from the training procedure of the optimal control and/or value function simplifies the theoretical study of our algorithms. It is nevertheless common to merge them in practice, as discussed in Remark 3.3.

Exploitation only strategy In the knowledge-based setting, there is no need for exhaustive and expensive (in time mainly) exploration of the state space, and the controller can directly choose training sets $\Gamma_{n}$ constructed from distributions $\mu_{n}$ that assign more points to the parts of the state space where the optimal process is likely to be driven. In practice, at time $n$, assuming we know that the optimal process is likely to stay in the ball centered around the point $m_{n}$ and with radius $r_{n}$, we choose a training measure $\mu_{n}$ centered around $m_{n}$ as, for example $\mathcal{N}(m_{n},r_{n}^{2})$, and build the training set as sample of the latter.

Explore first, exploit later If the agent has no idea of where to drive the process to receive large rewards, she can always proceed to an exploration step to discover favorable subsets of the state space. To do so, $\Gamma_{n}$, the training sets at time $n$, for $n=0,\ldots,N-1$, can be built by forward simulation using optimal strategies estimated using algorithms developed e.g. in [11]. We are then back to the Exploitation only strategy framework. An alternative suggestion is to use the approximate policy obtained from a first iteration of the hybrid-now algorithm, in order to derive the associated approximated optimal state process, $\hat{X}^{(1)}$. In a second iteration of the hybrid-now algorithm, we then use the marginal distribution $\mu_{n}^{(1)}$ of $\hat{X}_{n}^{(1)}$. We can even iterate this procedure in order to improve the choice of the training measure $\mu_{n}$, as proposed in [19].

Remark 3.3

*For stationary control problems, which are the common framework in reinforcement learning, it is usual to combine exploration of the (state and action) space and learning of the value function by drawing the mini-batches under the law given by the current estimate of the $Q$-value plus some noise (commonly called $\varepsilon$-greedy policy). In practice, this method works well, and offers a flexible trade-off between exploration and exploitation. It looks challenging, however, to study its performance theoretically because it mixes the gradient-descent steps for the training of the value function and the optimal control. Also, note that this method is not easily adaptable to finite-horizon control problems. *

4 Convergence analysis

This section is devoted to the convergence of the estimator $\hat{V}_{n}^{M}$ of the value function $V_{n}$ obtained from a training sample of size $M$ and using DNN algorithms listed in Section 3.

Training samples rely on a given family of probability distributions $\mu_{n}$ on ${\cal X}$, for $n=0,\ldots,N$, referred to as training distribution (see Section 3.3 for a discussion on the choice of $\mu$). For the sake of simplicity, we consider that $\mu_{n}$ does not depend on $n$, and denote then by $\mu$ the training distribution. We shall assume that the family of controlled transition probabilities has a density w.r.t. $\mu$, i.e.,

[TABLE]

We shall assume that $r$ is uniformly bounded in $(x,x^{\prime},a)$ $\in$ ${\cal X}^{2}\times\mathbb{A}$, and uniformly Lipschitz w.r.t. $(x,a)$, i.e.,

(Hd) There exists some positive constants $\|r\|_{\infty}$ and $[r]_{{}_{L}}$ s.t.

[TABLE]

Remark 4.1

Assumption 4 is usually satisfied when the state and control space are compacts. While the compactness on the control space $\mathbb{A}$ is not explicitly assumed, the compactness condition on the state space ${\cal X}$ turns out to be more crucial for deriving estimates on the estimation error (see Lemma 4.10), and will be assumed to hold true for simplicity. Actually, this compactness condition on ${\cal X}$ can be relaxed by truncation and localization arguments (see proposition A.1 in Appendix A) by considering a training distribution $\mu$ such that 4 is true and which admits a moment of order 1, i.e. $\int|y|d\mu(y)$ $<$ $+\infty$. $\Box$

We shall also assume some boundedness and Lipschitz condition on the reward functions:

(HR) There exists some positive constants $\|f\|_{\infty}$, $\|g\|_{\infty}$, $[f]_{{}_{L}}$, and $[g]_{{}_{L}}$ s.t.

[TABLE]

Under this boundedness condition, it is clear that the value function $V_{n}$ is also bounded, and we have: $\lVert V_{n}\rVert_{\infty}\leq(N-n)\lVert f\rVert_{\infty}+\lVert g\rVert_{\infty},$ for $n\in\{0,...,N\}.$

We shall finally assume a Lipschitz condition on the dynamics of the MDP.

(HF) For any $e$ $\in$ $E$, there exists $C(e)$ such that for all couples $(x,a)$ and $(x^{\prime},a^{\prime})$ in ${\cal X}\times\mathbb{A}$:

[TABLE]

For $M$ $\in$ $\mathbb{N}^{*}$, let us define $\rho_{M}=\mathbb{E}\left[\displaystyle\sup_{1\leq m\leq M}C(\varepsilon^{m})\right],$ where the $(\varepsilon^{m})_{m}$ is an i.i.d. sample of the noise $\varepsilon$. The rate of convergence of $\rho_{M}$ toward infinity will play a crucial role to show the convergence of the algorithms.

Remark 4.2

A typical example when 4 holds is the case where $F$ is defined through the time discretization of an Euler scheme, i.e. $F(x,a,\varepsilon):=b(x,a)+\sigma(x,a)\varepsilon,$ with $b$ and $\sigma$ Lipschitz-continuous w.r.t. the couple $(x,a)$, and $\varepsilon\sim\mathcal{N}(0,I_{d})$, where $I_{d}$ is the identity matrix of size $d\times d$. Indeed, in this case, it is straightforward to see that $C(\varepsilon)=[b]_{L}+[\sigma]_{L}\lVert\varepsilon\rVert_{d}$, where $[b]_{L}$ and $[\sigma]_{L}$ stand for the Lipschitz coefficients of $b$ and $\sigma$, and $\lVert.\rVert_{d}$ stands for the Euclidean norm in $\mathbb{R}^{d}$. Moreover, one can show that:

[TABLE]

which implies in particular that $\rho_{M}\underset{M\to+\infty}{=}\mathcal{O}\left(\sqrt{\log(M)}\right).$ Let us indeed check the inequality (9). For this, let us fix some integer $M^{\prime}>0$ and let $Z:=\displaystyle\sup_{1\leq m\leq M^{\prime}}|\epsilon_{1}^{m}|$ where $\epsilon_{1}^{m}$ are i.i.d. such that $\epsilon_{1}^{1}\sim\mathcal{N}(0,1)$. From Jentzen inequality to the r.v. $Z$ and the convex function $z\mapsto\exp(tz)$, where $t>0$ will be fixed later, we get

[TABLE]

where we used the closed-form expression of the moment generating function of the folded normal distributionaaaThe folded normal distribution is the one of $|Z|$ with $Z\sim\mathcal{N}_{1}(\mu,\sigma)$. Its moment generating function is given by $t\mapsto\exp\left(\frac{\sigma^{2}t^{2}}{2}+\mu t\right)\left[1-\Phi\left(-\frac{\mu}{\sigma}-\sigma t\right)\right]+\exp\left(\frac{\sigma^{2}t^{2}}{2}-\mu t\right)\left[1-\Phi\left(\frac{\mu}{\sigma}-\sigma t\right)\right]$, where $\Phi$ is the c.d.f. of $\mathcal{N}_{1}(0,1)$. to write the last inequality. Hence, we have for all $t>0$:

[TABLE]

We get, after taking $t=\sqrt{2\log(2M^{\prime})}$:

[TABLE]

Since inequality $\lVert x\rVert_{d}\leq d\lVert x\rVert_{\infty}$ holds for all $x\in\mathbb{R}^{d}$, we derive

[TABLE]

and apply (10) with $M^{\prime}=dM$, to complete the proof of (9).

$\Box$* *

Remark 4.3

Under 4, 4 and 4, it is straightforward to see from the dynamic programming formula (3) that $V_{n}$ is Lipschitz for all $n=0,\ldots,N$, with a Lipschitz coefficient $[V_{n}]_{L}$ that can be bounded by standard arguments as follows:

[TABLE]

using the usual convention $\frac{1-x^{p}}{1-x}=p$ for $p\in\mathbb{N}^{*}$ and $x=0$. The Lipschitz continuity of $V_{n}$ plays a significant role in proving the convergence of the Hybrid algorithms described and studied in section 4.2. * $\Box$*

4.1 Control learning by performance iteration (NNcontPI)

In this paragraph, we analyze the convergence of the NN control learning by performance iteration as described in Section 3.1. We shall consider the following set of neural networks for approximating the optimal policy:

[TABLE]

where $\lVert.\rVert$ is the Euclidean norm in $\mathbb{R}^{d}$. Note that the considered neural networks have one hidden layer, $K$ neurons, Relu activation functions, and $\sigma_{\mathbb{A}}$ as output layer. $\gamma$ and $\eta$ are often referred to in the literature as respectively total variation and kernel. The activation function $\sigma_{A}$ is chosen depending on the form of the control space $\mathbb{A}$. For example, when $\mathbb{A}$ $=$ $\mathbb{R}^{q}$, we simply take $\sigma_{A}$ as the identity function; when $\mathbb{A}$ $=$ $\mathbb{R}_{+}^{q}$, or more generally in the form $\mathbb{A}$ $=$ $\displaystyle\prod_{i=1}^{q}[a_{i},\infty)$, one can take the component-wise Relu activation function (possibly shifted and scaled); and when $\mathbb{A}$ $=$ $[0,1]^{q}$, or more generally $\mathbb{A}$ $=$ $\displaystyle\prod_{i=1}^{q}[a_{i},b_{i}]$, for $a_{i}$ $\leq$ $b_{i}$, $i$ $=$ $1,\ldots,q$, one can take the component-wise sigmoid activation function (possibly shifted and scaled); when $\mathbb{A}$ is a discrete space corresponding to discrete control values $\{a_{1},\cdots,a_{L}\}$, we may consider randomized control, hence valued in the simplex of $\hat{\mathbb{A}}$ of $[0,1]^{L}$, and use a softmax output layer activation function for parametrizing the discrete probabilities valued in $\hat{\mathbb{A}}$ (such algorithm, called ClassifPI, has been developed in [3]; see also [9] on softmax output for classification problems, and [1] about softmax policies for action selection and possible improvements.)

Let $K_{M}$, $\eta_{M}$ and $\gamma_{M}$ be sequences of integers such that

[TABLE]

We denote by $\mathcal{A}_{M}$ $:=$ $~{}^{\eta_{M}}\!\!\mathcal{A}_{K_{M}}^{\gamma_{M}}$ the class of neural network for policy as defined in (4.1) with parameters $\eta_{M}$, $K_{M}$ and $\gamma_{M}$ that satisfy the conditions (12).

Remark 4.4

In the case where $F$ is defined in dimension $d$ as: $F(x,a,\varepsilon)=b(x,a)+\sigma(x,a)\varepsilon,$ we can use (9) to get: $\rho_{M}^{N-n}\underset{M\to+\infty}{=}\mathcal{O}\left(\sqrt{\log(M)}^{N-n}\right).$ $\Box$* *

Recall that the approximation of the optimal policy in the NNcontPI algorithm is computed in backward induction as follows: For $n$ $=$ $N-1,\ldots,0$, generate a training sample for the state $X_{n}^{(m)}$, $m$ $=$ $1,\ldots,M$ from the training distribution $\mu$, and samples of the exogenous noise \big{(}\varepsilon_{k}^{m}\big{)}_{m=1,k=n+1}^{M,N}.

• •

Compute the approximated policy at time $n$

[TABLE]

where

[TABLE]

with \big{(}X^{(m),A}_{k}\big{)}_{k=n+1}^{N} defined by induction as follows, for m$$=$$1,\ldots,M:

[TABLE]

• •

Compute the estimated value function $\hat{V}_{n}^{M}$ as in (6).

Remark 4.5

Finding the $\underset{}{\mathrm{argmin\,}}$ in (13) is highly non-trivial and impossible in real situation. It is slowly approximated by (non stochastic) gradient-descent, i.e. with one large batch of size $M$, for which some convergence results are known under convexity or smoothness assumptions of $F$, $f$, $g$; and is approximated in practice by stochastic gradient-descent with a large enough number of epochs. We refer to [15] for error analysis of stochastic gradient descent optimization algorithms. In order to simplify the theoretical analysis, we assume that the $\underset{}{\mathrm{argmin\,}}$ in (13) is exactly reached. Our main focus here is to understand how the finite size of the training set affects the convergence of our algorithms. * $\Box$*

We now state our main result about the convergence of the NNcontPI algorithm.

Theorem 4.6

Assume that there exists an optimal feedback control $(a_{k}^{\mathrm{opt}})_{k=n}^{N-1}$ for the control problem with value function $V_{n}$, $n$ $=$ $0,\ldots,N$, and let $X_{n}\sim\mu$. Then, as $M\to\infty$bbbThe notation $x_{M}$ $=$ ${\cal O}(y_{M})$ as $M$ $\rightarrow$ $\infty$, means that the ratio $|x_{M}|/|y_{M}|$ is bounded as $M$ goes to infinity.

[TABLE]

where $\mathbb{E}$ stands for the expectation over the training set used to evaluate the approximated optimal policies $(\hat{a}_{k}^{M})_{n\leq k\leq N-1}$, as well as the path $(X_{n})_{n\leq k\leq N}$ controlled by the latter. Moreover, as $M\to\infty$cccThe notation $x_{M}$ $=$ ${\cal O}_{\mathbb{P}}(y_{M})$ as $M$ $\rightarrow$ $\infty$, means that there exists $c>0$ such that \mathbb{P}\big{(}|x_{M}|>c|y_{M}|\big{)}\to 0 as $M$ goes to infinity.

[TABLE]

*where $\mathbb{E}_{M}$ stands for the expectation conditioned by the training set used to estimate the optimal policies $(\hat{a}_{k}^{M})_{n\leq k\leq N-1}$. *

Remark 4.7

1. The term $\frac{\rho_{M}^{N-n-1}\gamma_{M}^{N-n-1}\eta_{M}^{N-n-2}}{\sqrt{M}}$ should be seen as the estimation error. It is due to the approximation of the optimal controls by means of neural networks in $\mathcal{A}_{M}$ using empirical cost functional in (13). We show in section B that this term disappears in the ideal case where the real cost functional (i.e. not the empirical one) is minimized.

2. The rate of convergence depends dramatically on $N$ since it becomes exponentially slower when $N$ goes to infinity. This is a huge drawback for this performance iteration-based algorithm. We will see in the next section that the rate of convergence of value iteration-based algorithms does not suffer from this problem. * $\Box$*

Comment: Since we clearly have $V_{n}$ $\leq$ $\hat{V}_{n}^{M}$, estimation (15) implies the convergence in $L^{1}$ norm of the NNcontPI algorithm, under condition (12), and in the case where

[TABLE]

This is actually the case under some regularity assumptions on the optimal controls, as stated in the following proposition.

Proposition 4.8

The two following assertions hold:

Assume that $a^{\mathrm{opt}}_{k}\in\mathbb{L}^{1}(\mu)$ for $k=n,...,N-1$. Then

[TABLE] 2. 2.

Assume that the function $a^{\mathrm{opt}}_{k}$ is $c$-Lipschitz for $k=n,...,N-1$. Then

[TABLE]

Proof. The first statement of Proposition 4.8 relies essentially on the universal approximation theorem, and the second assertion is stated and proved in [2]. For the sake of completeness, we provide more details in Appendix E. $\Box$

Remark 4.9

*The consistency of NNContPI is proved in Theorem 4.6, but no results on its variance are provided. We refer e.g. to Section 3.2 in [3] to observe the small variance of NNContPI for a linear quadratic problem in dimension up to 100. *

The rest of this section is devoted to the proof of Theorem 4.6. Let us introduce some useful notations. Denote by $\mathbb{A}^{{\cal X}}$ the set of Borelian functions from the state space ${\cal X}$ into the control space $\mathbb{A}$. For $n$ $=$ $0,\ldots,N-1$, and given a feedback control (policy) represented by a sequence $(A_{k})_{k=n,\ldots,N-1}$, with $A_{k}$ in $\mathbb{A}^{{\cal X}}$, we denote by $J_{n}^{(A_{k})_{k=n}^{N-1}}$ the cost functional associated with the policy $(A_{k})_{k}$. Notice that with this notation, we have $\hat{V}_{n}^{M}$ $=$ $J_{n}^{(\hat{a}_{k}^{M})_{k=n}^{N-1}}$. We define the estimation error at time $n$ associated with the NNContPI algorithm by

[TABLE]

with $X_{n}$ $\sim$ $\mu$: It measures how well the chosen estimator (e.g. mean square estimate) can approximate a certain quantity (e.g. the conditional expectation). Of course we expect the latter to cancel when the size of the training set used to build the estimator goes to infinity. Actually, we have

Lemma 4.10

For $n$ $=$ $0,\ldots,N-1$, the following holds:

[TABLE]

This implies in particular that

[TABLE]

*where we recall that $\rho_{M}=\mathbb{E}\left[\displaystyle\sup_{1\leq m\leq M}C(\varepsilon^{m})\right]$ is defined in 4. *

Proof. The relation (19) states that the estimation error cancels when $M\to\infty$ with a rate of convergence of order \mathcal{O}\Big{(}\frac{\rho_{M}^{N-n-1}\gamma_{M}^{N-n-1}\eta_{M}^{N-n-2}}{\sqrt{M}}\Big{)}. The proof is in the spirit of the one that can be found in chapter 9 of [10]. It relies on a technique of symmetrization by a ghost sample, and a wise introduction of additional randomness by random signs. The details are postponed to Section C in the Appendix. The proof of (20) follows from (19) by a direct application of the Markov inequality. $\Box$

Let us also define the approximation error at time $n$ associated with the NNContPI algorithm by

[TABLE]

where we recall that $\mathbb{E}_{M}$ denotes the expectation conditioned by the training set used to compute the estimates $(\hat{a}_{k}^{M})_{k=n+1}^{N-1}$ and the one of $X_{n}$ $\sim$ $\mu$.

$\varepsilon_{\mathrm{PI},n}^{\mathrm{approx}}$ measures how well the regression function can be approximated by means of neural networks functions in $\mathcal{A}_{M}$ (notice that the class of neural networks is not dense in the set $\mathbb{A}^{{\cal X}}$ of all Borelian functions).

Lemma 4.11

For $n$ $=$ $0,\ldots,N-1$, the following holds as $M\to\infty$,

[TABLE]

This implies in particular

[TABLE]

Proof. See Appendix D for the proof of (22). (4.11) is a direct application of the Markov inequality. $\Box$

Proof of Theorem 4.6.

Step 1. Let us denote by \hat{J}_{n,M}^{A,(\hat{a}^{M}_{k})_{k=n+1}^{N-1}}:=\frac{1}{M}\sum_{m=1}^{M}\Big{[}f\big{(}X_{n}^{(m)},A(X_{n}^{(m)})\big{)}+\hat{Y}_{n+1}^{(m),A}\Big{]}, the empirical cost function, from time $n$ to $N$, associated with the sequence of controls $(A,(\hat{a}^{M}_{k})_{k=n+1}^{N-1})$ and the training set, where we recall that $\hat{Y}_{n+1}^{(m),A}$ is defined in (14). We then have

[TABLE]

by definition of $\hat{V}_{n}^{M}$ and $\varepsilon_{\mathrm{PI},n}^{\mathrm{esti}}$. Moreover, for any $A\in\mathcal{A}_{M}$,

[TABLE]

Recalling that $\hat{a}_{n}^{M}=\underset{A\in\mathcal{A}_{M}}{\mathrm{argmin\,}}\hat{J}_{n,M}^{A,(\hat{a}^{M}_{k})_{k=n+1}^{N-1}},$ and taking the infimum over $\mathcal{A}_{M}$ in the l.h.s. of (4.1) first, and in the r.h.s. second, we then get

[TABLE]

Plugging this last inequality into (4.1) yields the following estimate

[TABLE]

Step 2. By definition (21) of the approximation error, using the law of iterated conditional expectations for $J_{n}$, and the dynamic programming principle for $V_{n}$ with the optimal control $a_{n}^{\mathrm{opt}}$ at time $n$, we have

[TABLE]

where $\mathbb{E}_{n}^{A}[.]$ stands for the expectation conditioned by $X_{n}$ at time $n$ and the training set, when strategy $A$ is followed at time $n$. Under the bounded density assumption in 4, we then get

[TABLE]

Step 3. From (26) and (4.1), we have

[TABLE]

By induction, this implies

[TABLE]

Use the estimations (20) for $\varepsilon_{\mathrm{PI},n}^{\mathrm{esti}}$ in Lemma 4.10, and (4.11) for $\varepsilon_{\mathrm{PI},n}^{\mathrm{approx}}$ in Lemma 4.11, and observe that $\hat{V}_{n}(X_{n})$ $\geq$ $V_{n}(X_{n})$ holds a.s., to complete the proof of (16). Finally, the proof of (15) is obtained by taking expectation in (4.1), and using estimations (19) and (22). $\Box$

4.2 Hybrid-Now algorithm

In this paragraph, we analyze the convergence of the hybrid-now algorithm. We shall consider the following set of neural networks for the value function approximation:

[TABLE]

Let $\eta_{M}$, $K_{M}$ and $\gamma_{M}$ be integers such that:

[TABLE]

with $\rho_{M}$ defined in 4.

In what follows we denote by $\mathcal{V}_{M}:=~{}^{\eta_{M}}\!\!\mathcal{V}_{K_{M}}^{\gamma_{M}}$ the space of neural networks for the estimated value functions at time $n$ $=$ $0,\ldots,N-1$, parametrized by the values $\eta_{M}$, $\gamma_{M}$ and $K_{M}$ that satisfy (29). We also consider the class ${\cal A}_{M}$ of neural networks for estimated feedback optimal control at time $n$ $=$ $0,\ldots,N-1$, as described in Section 4.1, with the same parameters $\eta_{M}$, $\gamma_{M}$ and $K_{M}$.

Recall that the approximation of the value function and optimal policy in the hybrid-now algorithm is computed in backward induction as follows:

• •

Initialize $\hat{V}_{N}^{M}$ $=$ $g$

• •

For $n$ $=$ $N-1,\ldots,0$, generate a training sample $X_{n}^{(m)}$, $m$ $=$ $1,\ldots,M$ from the training distribution $\mu$, and a training sample for the exogenous noise $\varepsilon_{n+1}^{(m)}$, $m$ $=$ $1,\ldots,M$.

• (i)

compute the approximated policy at time $n$

[TABLE]

where $X_{n+1}^{(m),A}$ $=$ $F(X_{n}^{(m)},A(X_{n}^{(m)}),\varepsilon_{n+1}^{(m)})$ $\sim$ $P^{A(X_{n}^{(m)})}(X_{n}^{(m)},dx^{\prime})$.

• (ii)

compute the untruncated estimation of the value function at time $n$

[TABLE]

and set the truncated estimated value function at time $n$

[TABLE]

Remark 4.12

Notice that we have truncated the estimated value function in (30) by an a priori bound on the true value function. This truncation step is natural from a practical implementation point of view, and is also used for simplifying the proof of the convergence of the algorithm. * $\Box$*

We now state our main result about the convergence of the Hybrid-Now algorithm.

Theorem 4.13

Assume that there exists an optimal feedback control $(a_{k}^{\mathrm{opt}})_{k=n}^{N-1}$ for the control problem with value function $V_{n}$, $n$ $=$ $0,\ldots,N$, and let $X_{n}\sim\mu$. Then, as ${M\rightarrow+\infty}$

[TABLE]

*where $\mathbb{E}_{M}$ stands for the expectation conditioned by the training set used to estimate the optimal policies $(\hat{a}_{k}^{M})_{n\leq k\leq N-1}$. *

Remark 4.14

*The consistency of Hybrid-Now is proved in Theorem 4.13, but no results on its variance are provided. The stability of Hybrid-Now has nevertheless been observed numerically in [3], and we refer e.g. to its Section 3.2 to observe the small variance of Hybrid-Now in a linear-quadratic problem in dimension up to 100. *

Comment: Theorem 4.13 states that the estimator for the value function provided by the hybrid-now algorithm converges in $\mathbb{L}^{1}(\mu)$ when the size of the training set goes to infinity. Note that $\left(\gamma_{M}^{4}\frac{K_{M}\log(M)}{M}\right)^{\frac{1}{2(N-n)}}$ and $\left(\gamma_{M}^{4}\frac{\rho_{M}^{2}\eta_{M}^{2}\log(M)}{M}\right)^{\frac{1}{4(N-n)}}$ stand for the estimation error made by estimating empirically respectively the value functions and the optimal control by neural networks; also: \displaystyle\sup_{n\leq k\leq N}\displaystyle\inf_{\Phi\in\mathcal{V}_{M}}\sqrt{\mathbb{E}\Big{[}|\Phi(X_{k})-V_{k}(X_{k})|^{2}\Big{]}} and \displaystyle\sup_{n\leq k\leq N}\displaystyle\inf_{A\in\mathcal{A}_{M}}\mathbb{E}\Big{[}|A(X_{k})-a^{\mathrm{opt}}_{k}(X_{k})|\Big{]}are the approximation error made by estimating respectively the value function and the optimal control by neural networks.

In order to prove Theorem 4.13, let us first introduce the estimation error at time $n$ associated with the Hybrid-Now algorithm by

[TABLE]

where \hat{Y}_{n+1}^{(m),A}=\hat{V}_{n+1}^{M}\big{(}X_{n+1}^{(m),A}\big{)}, and $X_{n+1}^{(m),A}=F\left(X_{n}^{(m)},A(X_{n}^{(m)}),\varepsilon_{n+1}^{m}\right)$. We have the following bound on this estimation error:

Lemma 4.15

For $n$ $=$ $0,...,N-1$, the following holds:

[TABLE]

Proof. See Appendix F. $\Box$

Remark 4.16

The result stated by lemma 4.15 is sharper than the one stated in Lemma 4.10 for the performance iteration procedure. The main reason is that we can make use of the $\gamma_{M}\eta_{M}$-Lipschitz-continuity of the estimate of the value function at time $n+1$. * $\Box$*

We next introduce the approximation error at time $n$ associated with the Hybrid Now algorithm by

[TABLE]

where $\hat{Y}_{n+1}^{A}:=\hat{V}_{n+1}^{M}\left(F\left(X_{n},A(X_{n}),\varepsilon_{n+1}\right)\right)$. We have the following bound on this approximation error:

Lemma 4.17

For $n$ $=$ $0,...,N-1$, the following holds:

[TABLE]

Proof. See Appendix G. $\Box$

Proof of Theorem 4.13

Observe that not only the optimal strategy but also the value function is estimated at each time step $n$ $=$ $0,...,N-1$ using neural networks in the hybrid algorithm. It spurs us to introduce the following auxiliary process $(\bar{V}_{n}^{M})_{n=0}^{N}$ defined by backward induction as:

[TABLE]

and we notice that for $n$ $=$ $0,...,N-1$, $\bar{V}_{n}^{M}$ is the quantity estimated by $\hat{V}_{n}^{M}$.

Step 1. We state the following estimates: for $n$ $=$ $0,...,N-1$,

[TABLE]

and,

[TABLE]

where $\mathbb{E}_{M,n,X_{n}}$ stands for the expectation conditioned by the training set and $X_{n}$.

Let us first show the estimate (34). Note that inequality

[TABLE]

holds because $\hat{a}_{n}^{M}$ cannot do better than the optimal strategy. Take its expectation to get the first inequality in (34). Moreover, we write

[TABLE]

which holds by the same arguments as those used to prove (26). We deduce that

[TABLE]

This completes the proof of the second inequality stated in (34).

On the other hand, noting:

[TABLE]

and using (34), we obtain the inequality (35).

Step 2. We state the following estimation: for all $n\in\{0,...,N\}$

[TABLE]

where {\lVert.\rVert_{M,p}=\Big{(}\mathbb{E}_{M}\left[|.|^{p}\right]}\Big{)}^{1\over p}, i.e. $\lVert.\rVert_{M,p}$ stands for the $\mathbb{L}^{p}$ norm conditioned by the training set, for $p\in\{1,2\}$. The proof relies on Lemma H.1 and Lemma H.3 (see Appendix H).

Let us first show the following relation:

[TABLE]

For this, take $\delta_{M}=\gamma_{M}^{4}K_{M}\frac{\log(M)}{M}$, let $\delta>\delta_{M}$, and denote

[TABLE]

Apply Lemma H.3 to obtain:

[TABLE]

where $\mathcal{N}_{2}(\varepsilon,\mathcal{V},x_{1}^{M})$ stands for the $\varepsilon$-covering number of $\mathcal{V}$ on $x_{1}^{M}$, which is introduced in section H, and where the last line holds since we assumed $\frac{M\delta_{M}}{\gamma_{M}^{2}}\xrightarrow[M\to 0]{}0$. Since $\delta$ $>$ $\delta_{M}:=\gamma_{M}^{4}K_{M}\frac{\log(M)}{M}$, we then have $\sqrt{\delta}\sqrt{K_{M}}\sqrt{\log(M)}\leq\frac{\sqrt{M}\delta}{\gamma_{M}^{2}}$, which implies that (74) holds by (4.2). Therefore, by application of Lemma H.1, the following holds:

[TABLE]

It remains to note that \mathbb{E}_{M}\Big{[}\big{|}\hat{V}_{n}^{M}(X_{n})-\bar{V}_{n}^{M}(X_{n})\big{|}^{2}\Big{]}\leq\mathbb{E}_{M}\Big{[}\big{|}\tilde{V}_{n}^{M}(X_{n})-\bar{V}_{n}^{M}(X_{n})\big{|}^{2}\Big{]} always holds, and this completes the proof of (37).

Next, let us show

[TABLE]

For this, take some arbitrary $\Phi\in\mathcal{V}_{M}$ and split

[TABLE]

To bound the last term in the r.h.s. of (40), we write

[TABLE]

Use the dynamic programming principle, assumption 4 and (35) to get:

[TABLE]

We then notice that

[TABLE]

holds a.s., so that

[TABLE]

and use Lemma 4.17 to bound $\varepsilon^{\mathrm{approx}}_{\mathrm{HN},n}$. By plugging into (40), and using the estimations in Lemmas 4.15 and 4.17, we obtain the estimate (39). Together with (37), this proves the required estimate(36). By induction, we get as $M\to\infty$,

[TABLE]

which completes the proof of Theorem 4.13. $\Box$

5 Conclusion

This paper develops new machine learning algorithms for high-dimensional Markov decision processes. We propose and compare two algorithms based on dynamic programming, performance/hybrid iteration and neural networks for approximating the control and value function. The main theoretical contribution is to provide a detailed convergence analysis for each of these algorithms: by using least squares neural network regression, we state error estimates in terms of the universal approximation error of neural networks, and of the statistical risk when estimating the network function. Numerical tests on various applications are presented in a companion paper [3].

Appendix A Localization

In this section, we show how to relax the boundedness condition on the state space by a localization argument.

Let $R>0$. Consider the localized state space $\bar{B}_{d}^{\cal X}(0,R):={\cal X}\cap\{\lVert x\rVert_{d}\leq R\}$, where $\lVert.\rVert_{d}$ is the Euclidean norm of $\mathbb{R}^{d}$. Let $\left(\bar{X}_{n}\right)_{0\leq n\leq N}$ be the Markov chain defined by its transition probabilities as

[TABLE]

for all Borelian $O$ in $\bar{B}_{d}^{\cal X}(0,R)$, where $\pi_{R}$ is the Euclidean projection of $\mathbb{R}^{d}$ on $\bar{B}_{d}^{\cal X}(0,R)$, and $\pi_{R}\circ\mu$ is the pushforward measure of $\mu$. Notice that the transition probability of $\bar{X}$ admits the same density $r$, for which 4 holds, w.r.t. $\pi_{R}\circ\mu$.

Define $(\bar{V}^{R}_{n})_{n}$ as the value function associated with the following stochastic control problem for $\left(\bar{X}_{n}\right)_{n=0}^{N}$:

[TABLE]

for $x\in\bar{B}_{d}^{\cal X}(0,R)$. By the dynamic programming principle, $(\bar{V}^{R}_{n})_{n}$ is solution of the following Bellman backward equation:

[TABLE]

We shall assume two conditions on the measure $\mu$.

(Hloc) $\mu$ is such that:

[TABLE]

Using the dominated convergence theorem, it is straightforward to see that A holds if $\mu$ admits a moment of order 1.

Proposition A.1

Let $X_{n}\sim\mu$. The following holds:

[TABLE]

where we denote $\lVert V\rVert_{\infty}=\displaystyle\sup_{0\leq k\leq N}\;\lVert V_{k}\rVert_{\infty}$, and use the convention $\frac{1-x^{p}}{1-x}=p$ for $x=0$ and $p>1$. Consequently, for all $n=0,...,N$, we get under A:

[TABLE]

**Comment: ** Proposition A.1 states that if ${\cal X}$ is not bounded, the control problem (43) associated with a bounded controlled process $\bar{X}$ can be as close as desired, in $\mathbb{L}^{1}(\mu)$ sense, to the original control problem by taking $R$ large enough. Moreover, as stated before, the transition probability of $\bar{X}$ admits the same density $r$ as $X$ w.r.t. the pushforward measure $\pi_{R}\circ\mu$.

Proof of Proposition A.1. Take $X_{n}\sim\mu$ and write:

[TABLE]

Let us first bound the first term in the r.h.s. of (A), by showing that, for $n=0,\ldots,N-1$:

[TABLE]

Take $x\in\bar{B}_{d}(0,R)$ and notice that

[TABLE]

It remains to inject this bound in the expectation to obtain (45).

To bound the second term in the r.h.s. of (A), notice that

[TABLE]

holds a.s., which implies:

[TABLE]

Plugging (45) and (46) into (A) yields:

[TABLE]

with $X_{n}$ and $X_{n+1}$ i.i.d. following the law $\mu$. The result stated in proposition A.1 then follows by induction. $\Box$

Appendix B Forward evaluation of the optimal controls in $\mathcal{A}_{M}$

We evaluate in this section the real performance of the best controls in $\mathcal{A}_{M}$. Let $(a^{\mathcal{A}_{M}}_{n})_{n=0}^{N-1}$ be the sequence of optimal controls in the class of neural networks $\mathcal{A}_{M}$, and denote by $(J^{\mathcal{A}_{M}}_{n})_{0\leq n\leq N}$ the cost functional sequence associated with $(a^{\mathcal{A}_{M}}_{n})_{n=0}^{N-1}$ and characterized as solution of the Bellman equation:

[TABLE]

where $\mathbb{E}_{n,X_{n}}^{A}[\cdot]$ stands for the expectation conditioned by $X_{n}$ and when the control $A$ is applied at time $n$.

In this section, we are interested in comparing $J^{\mathcal{A}_{M}}_{n}$ to $V_{n}$. Note that $V_{n}(x)\leq J^{\mathcal{A}_{M}}_{n}(x)$ holds for all $x\in{\cal X}$, since $\mathcal{A}_{M}$ is included in the set of the Borelian functions of ${\cal X}$. We can actually show the following:

Proposition B.1

Assume that there exists a sequence of optimal feedback controls $(a^{\mathrm{opt}}_{n})_{0\leq n\leq N-1}$ for the control problem with value function $V_{n}$, $n=0,\ldots,N$. Then the following holds, as $M\to\infty$:

[TABLE]

Remark B.2

Notice that there is no estimation error term in (47), since the optimal strategies in $\mathcal{A}_{M}$ are defined as those minimizing the real cost functionals in $\mathcal{A}_{M}$, and not the empirical ones. * $\Box$*

Proof of Proposition B.1. Let $n\in\{0,...,N-1\}$, and $X_{n}\sim\mu$. Take $A\in\mathcal{A}_{M}$, and denote $J^{A}_{n}(X_{n})=f(x,A(x))+\mathbb{E}_{n,X_{n}}^{A}[J^{\mathcal{A}_{M}}_{n+1}(X_{n+1})]$. Clearly, we have $J^{\mathcal{A}_{M}}_{n}=\displaystyle\min_{A\in\mathcal{A}_{M}}J^{A}_{n}$. Moreover:

[TABLE]

Applying assumption 4 to bound the last term in the r.h.s. of (48) yields

[TABLE]

which holds for all $A\in\mathcal{A}_{M}$, so that:

[TABLE]

(47) then follows directly by induction. $\Box$

Appendix C Proof of Lemma 4.10

The proof is divided into four steps.

Step 1: Symmetrization by a ghost sample. We take $\varepsilon>0$ and show that for

{M>2\frac{\big{(}(N-n)\lVert f\rVert_{\infty}+\lVert g\rVert_{\infty}\big{)}^{2}}{\varepsilon^{2}}}, the following holds:

[TABLE]

where:

• •

$(X^{{}^{\prime}(m)}_{k})_{1\leq m\leq M,n\leq k\leq N}$ is a copy of $(X^{(m)}_{k})_{1\leq m\leq M,n\leq k\leq N}$ generated from an independent copy of the exogenous noises $(\varepsilon^{\prime(m)}_{k})_{1\leq m\leq M,n\leq k\leq N}$, and independent copy of initial positions at time $n$, $(X^{{}^{\prime}(m)}_{n})_{m=1}^{M}$, following the same control $\hat{a}^{M}_{k}$ at time k$$=n+1,\dots,N-1, and control $A$ at time $n$,

• •

We recall that $Y^{(m),A}_{n+1}$ has already been defined in (14), and we similarly define

[TABLE]

Let $A^{*}\in\mathcal{A}_{M}$ be such that:

[TABLE]

if such a function exists, and an arbitrary function in $\mathcal{A}_{M}$ if such a function does not exist. Note that \frac{1}{M}\sum_{m=1}^{M}\Big{[}f(X_{n}^{(m)},A^{*}(X_{n}^{(m)}))+\hat{Y}_{n+1}^{(m),A^{*}}\Big{]}-\mathbb{E}\Big{[}J_{n}^{A^{*},(\hat{a}^{M}_{k})_{k=n+1}^{N-1}}(X_{n})\Big{]} is a r.v., which implies that $A^{*}$ also depends on $\omega\in\Omega$. Denote by $\mathbb{P}_{M}$ the probability conditioned by the training set of exogenous noises $(\varepsilon^{(m)}_{k})_{1\leq m\leq M,n\leq k\leq N}$ and initial positions $(X^{(m)}_{k})_{1\leq m\leq M,n\leq k\leq N}$, and recall that $\mathbb{E}_{M}$ stands for the expectation conditioned by the latter. An application of Chebyshev’s inequality yields

[TABLE]

where we have used 0\leq\Big{|}J_{n}^{A^{*},(\hat{a}^{M}_{k})_{k=n+1}^{N-1}}(X^{\prime}_{n})\Big{|}\leq(N-n)\lVert f\rVert_{\infty}+\lVert g\rVert_{\infty} which implies

[TABLE]

Thus, for M>2\frac{\big{(}(N-n)\lVert f\rVert_{\infty}+\lVert g\rVert_{\infty}\big{)}^{2}}{\varepsilon^{2}}, we have

[TABLE]

Hence:

[TABLE]

Observe that \frac{1}{M}\sum_{m=1}^{M}\Big{[}f(X_{n}^{(m)},A^{*}(X_{n}^{(m)}))+\hat{Y}^{(m),A^{*}}_{n+1}\Big{]}-\mathbb{E}_{M}\Big{[}J_{n}^{A^{*},(\hat{a}^{M}_{k})_{k=n+1}^{N-1}}(X_{n})\Big{]} is measurable w.r.t. the $\sigma$-algebra generated by the training set, so that conditioning by the training set and injecting (C) yields

[TABLE]

for M>2\frac{\big{(}(N-n)\lVert f\rVert_{\infty}+\lVert g\rVert_{\infty}\big{)}^{2}}{\varepsilon^{2}}, where we use the definition of $A^{*}$ to go from the second-to-last to the last line. The proof of (49) is then completed.

Step 2: We show that

[TABLE]

Indeed, let $M^{\prime}=\sqrt{2}\frac{(N-n)\lVert f\rVert_{\infty}+\lVert g\rVert_{\infty}}{\sqrt{M}}$, and notice

[TABLE]

The second term in the r.h.s. of (C) comes from (49). It remains to write the latter as an expectation to obtain (51).

*Step 3: Introduction of additional randomness by random signs.

*Let $(r_{m})_{1\leq m\leq M}$ be i.i.d. Rademacher r.v.dddThe probability mass function of a Rademacher r.v. is by definition $\frac{1}{2}\delta_{-1}+\frac{1}{2}\delta_{1}$.. We show that:

[TABLE]

Since for each $m=1,...,M$ the set of exogenous noises $(\varepsilon^{\prime(m)}_{k})_{n\leq k\leq N}$ and $(\varepsilon^{(m)}_{k})_{n\leq k\leq N}$ are i.i.d., their joint distribution remains the same if one randomly interchanges the corresponding components. Thus, the following holds for $\varepsilon$ $\geq$ [math]:

[TABLE]

It remains to integrate on $\mathbb{R}_{+}$ w.r.t. $\varepsilon$ to get (53).

Step 4: We show that

[TABLE]

Adding and removing the cost obtained by control 0 at time $n$ yields:

[TABLE]

We now bound the first term of the r.h.s. of (55). By the Cauchy-Schwarz inequality, and recalling that $(r_{m})_{1\leq m\leq M}$ are i.i.d. with zero mean such that $r_{m}^{2}=1$, we get

[TABLE]

Turn now to the second term of (55). By the Lipschitz continuity of $f$, it stands:

[TABLE]

where we condition by the exogenous noise, use assumption (HF-PI) and the $\eta_{M}\gamma_{M}$-Lipschitz continuity of the estimated optimal controls at time k, for $k=n+1,\ldots,N-1$. Now, notice first that

[TABLE]

and moreover:

[TABLE]

where $R>0$ is a bound for the state space (see e.g. the discussion on the Frank-Wolfe step p.10 of [2] for a proof of this inequality), which implies by the Cauchy-Schwarz inequality:

[TABLE]

since $(r_{m})_{m}$ are i.i.d. Rademacher r.v.. Plug (58) and (59) into (57) to obtain

[TABLE]

Plug then (56) and (60) into (55) to get (54).

Step 5: Conclusion. Plug (54) into (53) and combine it with (51) to obtain the bound on the estimation error, as stated in (19) of Lemma 4.10. $\Box$

Appendix D Proof of Lemma 4.11

Let $(\hat{a}^{M}_{k})_{k=n+1}^{N-1}$ be the sequence of estimated controls at time $k=n+1,...,N-1$. Take $A\in\mathcal{A}_{M}$ and recall that we denote by $J^{A,(\hat{a})_{k=n+1}^{N-1}}_{n}$ the cost functional associated with the control $A$ at time $n$, and $\hat{a}^{M}_{k}$ at time $k=n+1,\ldots,N-1$. The latter is characterized as solution of the Bellman equation

[TABLE]

where $\mathbb{E}_{n,x}^{A}[\cdot]$ stands for the expectation conditioned by $\{X_{n}=x\}$ when feedback control $A$ is followed at time $n$.

Take $n\in\{1,...,N\}$. The following holds:

[TABLE]

where the last inequality stands because the value function is smaller than the cost functional associated with any other strategy. We then apply the dynamic programming principle to obtain:

[TABLE]

To bound the r.h.s. of (62), first observe that for $A\in\mathcal{A}_{M}$:

[TABLE]

where we used twice assumption 4 at the second-last line of (63). Inject

[TABLE]

into (63) to obtain:

[TABLE]

Plugging (64) into (62) yields

[TABLE]

which implies by induction, as $M\to+\infty$:

[TABLE]

We now use Lemma 4.10 to bound the expectations of the $\varepsilon^{\mathrm{esti}}_{\mathrm{PI},k}$ for $k=n+1,\ldots,N-1$, and plug the result into (61) to complete the proof of Lemma 4.11. $\Box$

Appendix E Function approximation by neural networks

We assume $a^{\mathrm{opt}}_{k}\in\mathbb{L}^{2}(\mu)$, and show the relation (17) in Proposition 4.8.

The universal approximator theorem applies for $\mathcal{A}_{\infty}:=\bigcup_{M=1}^{\infty}\mathcal{A}_{M},$ and states that for all $\varepsilon>0$, for all $k$, there exists a neural network $a^{*}$ in $\mathcal{A}_{\infty}$ such that: $\lVert a^{\mathrm{opt}}_{k}-a^{*}\rVert_{\infty}<\frac{\varepsilon}{\mathcal{V}_{d}({\cal X})},$ where $\mathcal{V}_{d}({\cal X})$ stands for the volume of compact set ${\cal X}$ seen as a compact of the Euclidean space $\mathbb{R}^{d}$. By integrating, we then get: \sup_{n\leq k\leq N-1}\int_{{\cal X}}\big{|}a^{\mathrm{opt}}_{k}(x)-a^{*}(x)\big{|}d\mu(x)<{\varepsilon}. Also, notice that \big{(}\mathcal{A}_{M}\big{)}_{M\geq 1} is increasing, which implies that $\mathcal{A}_{\infty}=\lim_{M\to+\infty}\mathcal{A}_{M}$ and gives the existence of $M>0$ that depends on $\varepsilon$ such that $a^{*}\in\mathcal{A}_{M}$.

Therefore, we have shown that for $n=0,...,N-1$

[TABLE]

which is the required result stated in (17). $\Box$

We now show (18) of proposition 4.8. As stated in Proposition 6 of Section 4.7 in [2]: we can approximate a $c$-Lipschitz function by a $\gamma_{1}$-norm less than $\gamma_{M}$ and uniform error less than $c\left(\frac{\gamma_{M}}{c}\right)^{-2d/(d+1)}\log\frac{\gamma_{M}}{c}$, and proposition 1 in [2] shows that a function with $\gamma_{1}$ less than $\gamma_{M}$ may be approximated with $K_{M}$ neurons with uniform error $\gamma_{M}K_{M}^{-(d+3)/(2d)}$.

Thus, given $K_{M}$ and $\gamma_{M}$, there exists a neural network $a^{*}$ in $\mathcal{V}_{M}$ such that

[TABLE]

Appendix F Proof of Lemma 4.15

We prove Lemma 4.15 in four steps. Since the proof is very similar to the one of Lemma 4.10, we only detail the arguments that are modified.

Step 1: Symmetrization by a ghost sample. We take $\varepsilon>0$ and show that for M>2\frac{\big{(}(N-n)\lVert f\rVert_{\infty}+\lVert g\rVert_{\infty}\big{)}^{2}}{\varepsilon^{2}}, the following holds

[TABLE]

where \big{(}X^{\prime(m)}_{n}\big{)}_{m=1}^{M} is an i.i.d. copy of \big{(}X_{n}^{(m)}\big{)}_{m=1}^{M}; \big{(}\varepsilon^{\prime m}_{n+1}\big{)}_{m=1}^{M} is an i.i.d. copy of \big{(}\varepsilon^{m}_{n+1}\big{)}_{m=1}^{M}; \hat{Y}^{(m),A}_{n+1}:=\hat{V}_{n+1}^{M}\left(F\Big{(}X_{n}^{(m)},A\big{(}X_{n}^{(m)}\big{)},\varepsilon^{m}_{n+1}\Big{)}\right); and finally \hat{Y}^{\prime\,(m),A}_{n+1}:=\hat{V}_{n+1}^{M}\left(F\Big{(}X_{n}^{\prime(m)},A\big{(}X_{n}^{\prime(m)}\big{)},\varepsilon^{\prime m}_{n+1}\Big{)}\right).

Since $\hat{V}_{n}^{M}$ the estimated value function at time $n$, for n$$=$$0,...,N-1, is bounded by construction, (LABEL:ineq::step1_hybrid) is proved the same as in step 1 of Lemma 4.10. $\Box$

Step 2: The following result holds

[TABLE]

and it is proved the same way as step 2 in the proof of Lemma 4.10. $\Box$

Step 3: Introduction of additional randomness by random signs. The following result holds:

[TABLE]

as proved already in step 3 in the proof of Lemma 4.10. $\Box$

Step 4: We show that

[TABLE]

Adding and removing the cost obtained by control 0 at time $n$ yields:

[TABLE]

The first term in the r.h.s. in (70) is bounded as in the proof of Lemma 4.10 by $\frac{(N-n)\lVert f\rVert_{\infty}+\lVert g\rVert_{\infty}}{\sqrt{M}}.$ We use the Lipschitz-continuity of $f$ as follows, to bound its second term:

[TABLE]

where we conditioned by the exogenous noise, used assumption (HF), and the $\eta_{M}\gamma_{M}$-Lipschitz continuity of the estimated value fonction at time $n+1$.

By using the same arguments as those presented to prove Lemma 4.10, we have \mathbb{E}\bigg{[}\sup_{A\in\mathcal{A}_{M}}\bigg{|}\sum_{m=1}^{M}r_{m}A(X_{n}^{(m)})\bigg{|}\bigg{]}\leq\gamma_{M}\sqrt{M}, and then conclude that (69) holds.

*Step 5: Conclusion

*Combining(67),(68) and (69) results in the bound on the estimation error as stated in (32). $\Box$

Appendix G Proof of Lemma 4.17

We divide the proof of Lemma 4.17 into two steps. First write

[TABLE]

Step 1: We show

[TABLE]

Take $A\in\mathcal{A}_{M}$, and apply the dynamic programming principle to write

[TABLE]

where we used 4 at the second-to-last line. By using one more time assumption 4, we then get:

[TABLE]

which is the result stated in (72).

Step 2: We show

[TABLE]

Write

[TABLE]

which completes the proof of (73).

Step 3 *Conclusion:

*We complete the proof of Lemma 4.17 by plugging (72) and (73) into (G). $\Box$

Appendix H Some useful Lemmas for the proof of Theorem 4.13

Fix $M$ $\in$ $\mathbb{N}^{*}$, let $x_{1},\ldots,x_{M}\in\mathbb{R}^{d}$, and set $x^{M}=(x_{1},\ldots,x_{M})$. Define the distance $d_{2}(f,g)$ between $f:\mathbb{R}^{d}\to\mathbb{R}$ and $g:\mathbb{R}^{d}\to\mathbb{R}$ by

[TABLE]

An $\varepsilon$-cover of $\mathcal{V}$ is a set of functions $f_{1},\ldots,f_{P}:\mathbb{R}^{d}\to\mathbb{R}$ such that

[TABLE]

Let $\mathcal{N}_{2}(\varepsilon,\mathcal{V},x^{M})$ denote the size of the smallest $\varepsilon$-cover of $\mathcal{V}$ w.r.t. the distance $d_{2}$, and set by convention $\mathcal{N}_{2}(\varepsilon,\mathcal{V},x^{M})=+\infty$ if there does not exist any $\varepsilon$-cover of $\mathcal{V}$ of finite size. $\mathcal{N}_{2}(\varepsilon,\mathcal{V},x^{M})$ is called $\L^{2}$-$\varepsilon$-covering number of $\mathcal{V}$ on $x^{M}$.

Lemma H.1

*Let $(X,Y)$ be a random variable. Assume $|Y|\leq L$ a.s. and let $m(x)=\mathbb{E}[Y|X=x].$ Assume $Y-m(X)$ is sub-Gaussian in the sense that

\displaystyle\max_{m=1,...,M}c^{2}\mathbb{E}\Big{[}e^{(Y-m(X))^{2}/c^{2}}-1|X\Big{]}\leq\sigma^{2}\quad\text{ a.s.} for some $c,\sigma>0$. Let $\gamma_{M},L\geq 1$ and assume that the regression function is bounded by $L$ and that $\gamma_{M}\xrightarrow[M\to+\infty]{}+\infty$. Set*

[TABLE]

for some $\mathcal{V}_{M}$ of functions $\Phi:\mathbb{R}^{d}\to[-\gamma_{M},\gamma_{M}]$ and some random variables ${Y}_{1},...,{Y}_{M}$ which are bounded by $L$, and denote

[TABLE]

Then there exist constants $c_{1},c_{2}>0$ which depend only on $\sigma$ and $c$ such that for any $\delta_{M}>0$ with

[TABLE]

[TABLE]

for all $\delta\geq\delta_{M}$ and all $g\in\mathcal{V}_{M}\cup\{m\}$ we have as $M\to+\infty$:

[TABLE]

Proof H.2

*We refer to [16] (see its Theorem 3) for a proof. *

Lemma H.3

For any $\varepsilon$ $>$ [math], we have

[TABLE]

*where the class of neural networks $\mathcal{V}_{M}$ is defined in Section 4.2. *

Proof H.4

*We refer to [17] for a proof. *