Error bounds for approximations with deep ReLU neural networks in $W^{s,p}$ norms

TL;DR

This paper investigates how well deep ReLU neural networks can approximate Sobolev-regular functions in weaker Sobolev norms, providing both upper and lower bounds that are relevant for PDE numerical analysis.

Contribution

It introduces new approximation rates for ReLU networks in Sobolev spaces and establishes fundamental lower bounds, extending approximation theory in this context.

Findings

Constructed neural networks achieving specific approximation rates

Established lower bounds for ReLU network approximation in Sobolev norms

Extended approximation theory to PDE-relevant Sobolev regimes

Abstract

We analyze approximation rates of deep ReLU neural networks for Sobolev-regular functions with respect to weaker Sobolev norms. First, we construct, based on a calculus of ReLU networks, artificial neural networks with ReLU activation functions that achieve certain approximation rates. Second, we establish lower bounds for the approximation by ReLU neural networks for classes of Sobolev-regular functions. Our results extend recent advances in the approximation theory of ReLU networks to the regime that is most relevant for applications in the numerical analysis of partial differential equations.

Equations

$$T_l:{\mathbb{R}}^{N_{l-1}}\to {\mathbb{R}}^{N_l}$$

$$f_l:{\mathbb{R}}^{N_{l-1}}\to {\mathbb{R}}^{N_l},x\mapsto \varrho (T_l(x))$$

$$f_L:{\mathbb{R}}^{N_{L-1}}\to {\mathbb{R}}^{N_L},x\mapsto T_L(x).$$

$$\varrho :\mathbb{R}\to \mathbb{R},x\mapsto \max \{0,x\}.$$

$$\frac{1}{m}\sum _{i=1}^{m}l\left(\mathcal{N}(x_i|w),f(x_i)\right)$$

$$\frac{1}{m}\sum _{i=1}^m\left[l\left(\mathcal{N}(x_i|w),f(x_i)\right)+\sum _{j=1}^k{l}_j\left(D_x^j\mathcal{N}(x_i|w),D_x^{j}f(x_i)\right)\right],$$

$$\Vert {\mathcal{N}}_{\epsilon }(\cdot |w_f)-f(\cdot ){\Vert }_{L^{\infty }({\left(0,1\right)}^d)}\le \epsilon ,$$

$$\Vert {\mathcal{N}}_{\epsilon }(\cdot |w_f)-f(\cdot ){\Vert }_{W^{s,p}({\left(0,1\right)}^d)}\le \epsilon ,$$

$$\Vert A{\Vert }_{{\ell }^0}:=\left|\{(i,j):A_{i,j}\ne 0\}\right|.$$

$$\left[\begin{array}[]{c c}A&B\end{array}\right]\in\mathbb{R}^{d_{1},d_{2}+d_{3}}\quad\text{or}\quad\left[\begin{array}[]{c|c}A&B\end{array}\right]\in\mathbb{R}^{d_{1},d_{2}+d_{3}}$$

$$x^{\alpha }:=\prod _{i=1}^d{x}_i^{{\alpha }_i}.$$

$$C_c^{\infty }(\Omega ):=\left\{f\in C^{\infty }(\Omega )|\mathrm{supp}\text{leavevmode}f\subset \subset \Omega \right\}.$$

$$D^{\alpha }f:=\frac{{\partial }^{|\alpha |}f}{\partial x_1^{{\alpha }_1}\partial x_2^{{\alpha }_2}\cdots \partial x_d^{{\alpha }_d}}.$$

$$|f(x)-f(y)|\le L|x-y|$$

$$C_1\Vert x{\Vert }_1\le \Vert x{\Vert }_2\le C_2\Vert x{\Vert }_1\text{for all }x\in X.$$

$$\Vert T{\Vert }_{\mathcal{L}(X,Y)}:=\sup \left\{\Vert Tx{\Vert }_Y:x\in X,\Vert x{\Vert }_X\le 1\right\}.$$

$$\Phi =((A_1,b_1),(A_2,b_2),\dots ,(A_L,b_L)),$$

$$R_{\varrho }(\Phi )(x)=x_L,$$

$$x_0:=x,$$

$$\displaystyle x_{l}:=\varrho\left(A_{l}\left[\begin{array}[]{c|c|c}x_{0}^{T}&\ldots&x_{l-1}^{T}\end{array}\right]^{T}+b_{l}\right),\quad\text{for }l=1,\ldots L-1,$$

$$\displaystyle x_{L}:=A_{L}\left[\begin{array}[]{c|c|c}x_{0}^{T}&\ldots&x_{L-1}^{T}\end{array}\right]^{T}+b_{L},$$

$$A_{l}=\left[\begin{array}[]{c|c|c}A_{l,x_{0}}&\ldots&A_{l,x_{l-1}}\end{array}\right],$$

$$x_l=\varrho \left(A_{l,x_0}{x}_0+\dots +A_{l,x_{l-1}}{x}_{l-1}+b_l\right),\text{for }l=1,\dots L-1,$$

$$x_L=A_{L,x_0}{x}_0+\dots +A_{L,x_{L-1}}{x}_{L-1}+b_L.$$

$$A_{l}=\left[\begin{array}[]{c|c|c}A_{l,x_{0}}&\ldots&A_{l,x_{l-1}}\end{array}\right],$$

$$x_0:=x,$$

$$x_l:=\varrho \left(A_{l,x_{l-1}}{x}_{l-1}+b_l\right),\text{for }l=1,\dots L-1,$$

$$x_L:=A_{L,x_{L-1}}{x}_{L-1}+b_L.$$

$$\mathcal{A}=((A_1,b_1),(A_2,b_2),\dots ,(A_L,b_L)),$$

$$\varrho :\mathbb{R}\to \mathbb{R},x\mapsto \max (0,x),$$

$${\Phi }^1=((A_1^1,b_1^1),\dots ,(A_L^1,b_L^1)),{\Phi }^2=((A_1^2,b_1^2),\dots ,(A_L^2,b_L^2))$$

$${\Phi }^1\bullet {\Phi }^2:=\left((A_1^2,b_1^2),\dots ,(A_{L_2-1}^2,b_{L_2-1}^2),({\tilde{A}}_1^1,{\tilde{b}}_1^1),\dots ,({\tilde{A}}_{L_1}^1,{\tilde{b}}_{L_1}^1)\right),$$

$$\tilde{A}_{l}^{1}:=\left[\begin{array}[]{c|c|c|c}A_{l,x_{0}}^{1}A_{L_{2}}^{2}&A_{l,x_{1}}^{1}&\dots&A_{l,x_{l-1}}^{1}\end{array}\right]\quad\text{and}\quad\tilde{b}_{l}^{1}:=A_{l,x_{0}}^{1}b_{L_{2}}^{2}+b_{l}^{1},$$

Full text

1 Introduction

Powered by modern, highly parallelized hardware and the immense amount of accessible data, deep neural networks substantially outperform both traditional modelling approaches, e.g. those based on differential equations, and classical machine learning methods in a wide range of applications. Prominent areas of application include image classification [33, 53, 35], speech recognition [30, 17, 60], and natural language processing [63].

Despite this overwhelming success in applications, a comprehensive mathematical explanation of this success is has not been found. However, a deep theoretical understanding of these techniques is crucial to design more efficient architectures and essential in safety-critical applications, such as autonomous driving.

Many attempts at unraveling the extreme efficiency of deep neural networks have been made in the context of approximation theory. The universal approximation theorem [15, 32] establishes that every continuous function on a compact domain can be uniformly approximated by neural networks. More refined results that also set into relation the size and the approximation fidelity of neural networks have been reported for smooth activation functions in, for example, [5, 41, 52, 9].

In most applications, the activation functions are not smooth but taken to be the piecewise linear ReLU function. For these networks, approximation bounds for classes of smooth functions have been established in [61] and for piecewise smooth functions in [47]. Connections to sparse grids and the associated approximation rates were established in [43] and connections to linear finite element approximation were reported in [27]. It was also discovered that the depth of neural networks, i.e., the number of layers, crucially influences the approximation capabilities of these networks in the sense that deeper networks are more efficient approximators [50, 42, 47, 13].

One of the applications of deep learning where a strong knowledge of the approximation capabilities of neural networks directly translates into quantifiable theoretical results appears when solving partial differential equations using deep learning techniques. Some notable advances in this direction have been made in [36, 59, 26, 54, 58, 19, 6]. In this regard, not only the approximation fidelity with respect to standard Lebesgue spaces is of high interest, but also that with respect to Sobolev-type norms. First results in this direction were reported in [51, 45].

In this work, we give a comprehensive analysis of the approximation rates of deep neural networks of Sobolev-regular functions with respect to (fractional) Sobolev norms.

In the remaining part of this introduction, we briefly describe neural network architectures relevant to this work. Then, we present the setting that neural networks are typically used in. Afterwards, we give a practical and theoretical motivation for studying approximations of functions and their derivatives with neural networks. Finally, we state our results.

1.1 Neural networks

There exists a wide variety of neural network architectures, each adapted to specific tasks. One of the most commonly-used architectures is a feedforward architecture also known as multi-layer perceptron which implements a function as a sequence of affine-linear transformations followed by a componentwise application of a non-linear function, called activation function. The length $L$ of the sequence is referred to as the number of layers of the network and the input of a layer consists of the output of the preceding layer. We denote by $N_{0}\in\mathbb{N}$ the dimension of the input space and by $N_{l}\in\mathbb{N}$ the dimension of the output of the $l$-th layer for $l=1,\ldots,L$. Thus, if $\varrho:\mathbb{R}\to\mathbb{R}$ is the activation function and

[TABLE]

is the affine-linear transformation of the $l$-th layer, then the computation in that layer can be described as

[TABLE]

for $l=1,\ldots,L-1$, and

[TABLE]

Note that $\varrho$ acts componentwise in all but the last layer and in the last layer no activation function is applied. The parameters defining the affine-linear transformations $T_{l}$ are referred to as weights and $\sum_{k=0}^{L}N_{l}$ is called the number of neurons of the network, since each output coordinate of $f_{l}$ can be seen as a small computational unit, similar to neurons in the brain. If a network has $3$ or more layers, then it is usually called deep and a $2$-layer network is called shallow. The complexity of a neural network is typically measured in the number of layers, weights, and neurons (see [3]). The term deep learning refers to the subset of machine learning methods associated with deep neural networks.

Recently, more general feedforward architectures which allow connections between non-neighboring layers, so-called skip connections, have been shown to yield state-of-the-art results in object recognition tasks [33, 28]. Here, the input of a layer consists of the output of all preceding layers. Note, that the case where no skip connections are allowed, is a special case of this more general architecture.

One of today’s most widely-used activation functions is the Rectified Linear Unit (ReLU) (see [37]), defined as

[TABLE]

The popularity of the ReLU can be explained by a number of factors. It is cheap to compute, promotes sparsity in data representation [7], alleviates the problem of vanishing-gradients [22], and thus yields better optimization properties. Moreover, the ReLU is the only commonly-used activation function such that the associated function spaces are not necessary non-closed [46]. In this work, we will mostly focus on ReLU networks with a feedforward architecture allowing skip connections.

1.2 Supervised learning with neural networks

Typically, neural networks are applied in supervised learning problems. The starting point is a dataset of input-output pairs $(x_{i},f(x_{i}))_{i=1}^{m}$, called samples, where $f$ is in most cases an unknown function111We will later see a case where $f$ is in fact known. with values only given at sample points $x_{i}$. As an example, $x_{i}$ can be thought of as an image and $f(x_{i})$ as a vector of scores, where each score is the probability of a certain category, e.g. ”dog” or ”cat”, being associated with $x_{i}$. During training, one then seeks to learn the function $f$ by adapting the weights $w$ of a neural network $\mathcal{N}$ such that the empirical loss

[TABLE]

is minimized for some loss function $l$. Ultimately, one is interested in how well the learning algorithm performs on formerly unseen data points $x$. The associated error is called the generalization error.

1.3 Motivation: approximating functions and derivatives

From a learning theory point of view, the task of estimating the generalization error decomposes into a statistical problem depending on the samples and an approximation theoretical problem, independent of the samples. An accessible introduction to learning theory from the perspective of approximation theory can be found in [14]. The aim of this work is to study the simultaneous approximation of a function and its derivative with a neural network. There are multiple scenarios in which this is possible and useful.

• •

In [16], the $j$-th order derivatives of $f$ are incorporated in the empirical loss function (1.1), resulting in an empirical loss

[TABLE]

which encourages the network to encode information about the derivatives of $f$ in its weights. The authors of [16] call this method Sobolev training and reported reduced generalization errors and better data-efficiency in a network compression task (see [31]) and in application to synthetic gradients (see [34]). In case of network compression, the approximated function $f$ is a function realized by a possibly very large neural network $\mathcal{N}_{\text{large}}(\cdot|w)$, that has been trained for some supervised learning task and is learnt by a smaller network $\mathcal{N}_{\text{small}}$. In contrast to usual supervised learning settings, the approximated function $f(\cdot)=\mathcal{N}_{\text{large}}(\cdot|w)$ is known and the derivatives can be computed.

• •

Motivated by the performance of deep learning-based solutions in classical machine learning tasks and, in particular, by their ability to overcome the curse of dimension, neural networks are now also applied for the approximative solution of partial differential equations (PDEs) (see [36, 59, 26, 54]).

In [54] the authors present their deep Galerkin method for approximating solutions of high-dimensional quasilinear parabolic PDEs. For this, a functional $J(f)$ encoding the differential operator, boundary conditions, and initial conditions is introduced. A neural network $\mathcal{N}_{\text{PDE}}$ with weights $w$ is then trained to minimize the functional $J(\mathcal{N}_{\text{PDE}}(w))$. This is done by a discretization and randomly sampling spatial points.

The theoretical foundation for approximating a function and higher-order derivatives with a neural network was already given in a less known version of the universal approximation theorem by Hornik in [32, Theorem 3]. In particular, it was shown that if the activation function $\varrho$ is $k$-times continuously differentiable, non-constant, and bounded, then any $k$-times continuously differentiable function $f$ and its derivatives up to order $k$ can be uniformly approximated by a shallow neural network on compact sets. Note though that the conditions on the activation function are very restrictive and that, for example, the ReLU is not included in the above result. However, in [16], it was shown that the theorem also holds for shallow ReLU networks if $k=1$. Theorem 3 in [32] was also used in [54] to show the existence of a shallow network approximating solutions of the PDEs considered in this paper.

An important aspect, that is untouched by the previous approximation results is how the complexity of a network and, in particular, its depth relates to its approximation properties.

1.4 Our contribution

In Theorem 1 in [61], Yarotsky showed upper complexity bounds for approximations in $L^{\infty}$ norm of functions from the Sobolev space $W^{n,\infty}(\left(0,1\right)^{d})$ with neural networks for continuous piecewise linear activation functions with a finite number of breakpoints. Precisely, it is shown there, that for any $\varepsilon>0$ there exists a neural network $\mathcal{N}_{\varepsilon}$ with at most $c\cdot\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ layers and at most $c\cdot\varepsilon^{-d/n}\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ weights and neurons such that for any $f\in W^{n,\infty}(\left(0,1\right)^{d})$ with $\lVert f\rVert_{W^{n,\infty}(\left(0,1\right)^{d})}\leq 1$ there is a choice of weights $w_{f}$ with

[TABLE]

where $c$ is a constant depending on $d$ and $n$.

Furthermore, under the assumption of a possibly discontinuous weight selection the number of weights needed by a neural network $\mathcal{N}_{\varepsilon}$ to be able to realize an $\varepsilon$-approximation in $L^{\infty}$ norm for any $f\in W^{n,\infty}(\left(0,1\right)^{d})$ with $\lVert f\rVert_{W^{n,\infty}(\left(0,1\right)^{d})}\leq 1$ is shown to be lower bounded by $c^{\prime}\cdot\varepsilon^{d/(2n)}$ in [61, Theorem 4 a)]. The constant $c^{\prime}$ depends on $d$ and $n$.

We show for the same set of activation functions that the approximation can also be done with respect to higher-order Sobolev norms with arbitrary $1\leq p\leq\infty$ and that there is a trade-off between the regularity used in the approximation norm and the regularity used in the complexity bounds. Specifically, we show that for any approximation accuracy $\varepsilon>0$ and regularity $0\leq s\leq 1$, there is a ReLU neural network $\mathcal{N}_{\varepsilon}$ with at most $c\cdot\log_{2}(\varepsilon^{-n/(n-s)})$ layers and $c\cdot\varepsilon^{-d/(n-s)}\cdot\log_{2}(\varepsilon^{-n/(n-s)})$ weights and neurons such that and any $f\in{W^{n,p}(\left(0,1\right)^{d})}$ with $\lVert f\rVert_{{W^{n,p}(\left(0,1\right)^{d})}}\leq B$ there is a choice of weights $w_{f}$ with

[TABLE]

where $c$ is a constant depending on $d,n,p,s$, and $B$. In the boundary case $s=0$ and $p=\infty$ our results corresponds to the theorem shown by Yarotsky and for $s=1$ and $p=\infty$ the function $\mathcal{N}_{\varepsilon}(\cdot|w_{f})$ and its weak gradient uniformly approximate $f$ and the weak gradient of $f$, respectively. For non-integer $s$ the function $\mathcal{N}_{\varepsilon}(\cdot|w_{f})$ approximates the function $f$ and its fractional derivative of order $s$ approximates the fractional derivative of order $s$ of $f$. The case $0\leq s\leq 1$ and $p=\infty$ was already shown by one of the authors, I. Gühring, in his Master thesis [25].

Moreover, analogously to [61, Theorem 4 a)] we establish lower bounds where the same regularity-complexity trade-off can be observed. We show that if a neural network $\mathcal{N}_{\varepsilon}$ is able to realize an $\varepsilon$-approximation in $W^{1,\infty}$ norm for any $f\in W^{n,\infty}(\left(0,1\right)^{d})$ with $\lVert f\rVert_{W^{n,\infty}(\left(0,1\right)^{d})}\leq 1$, then the number of weights of $\mathcal{N}_{\varepsilon}$ is lower bounded by $c^{\prime}\cdot\varepsilon^{\nicefrac{{-d}}{{2(n-1)}}}$. Here, $c^{\prime}$ is a constant depending on $d$ and $n$.

1.5 Outline

As a preparation, we start by introducing notation and some definitions in Section 1.6. In Section 2, we rigorously define a neural network and its architecture in mathematical terms and develop a network calculus. In Section 3, we briefly introduce (fractional) Sobolev spaces. In Section 4, we present our results which will be discussed in Section 5. To allow a concise presentation of our results most proofs are given in the appendix: Appendix A and B contain the necessary preparation for the proof of the results from Section 4.1 which is given in Appendix C. The results from Section 4.2 are proven in Appendix D.

1.6 Notation

We set $\mathbb{N}:=\{1,2,\ldots\}$ and $\mathbb{N}_{0}:=\mathbb{N}\cup\{0\}$. For $k\in\mathbb{N}_{0}$ the set $\mathbb{N}_{\geq k}:=\{k,k+1,\ldots\}$ consists of all natural numbers larger than or equal to $k$. For a set $A$ we denote its cardinality by $\lvert A\rvert\in\mathbb{N}\cup\{\infty\}$. If $x\in\mathbb{R}$, then we write $\lceil x\rceil:=\min\{k\in\mathbb{Z}:k\geq x\}$ where $\mathbb{Z}$ is the set of integers.

If $d\in\mathbb{N}$ and $\lVert\cdot\rVert$ is a norm on $\mathbb{R}^{d}$, then we denote for $x\in\mathbb{R}^{d}$ and $r>0$ by $B_{r,\lVert\cdot\rVert}(x)$ the open ball around $x$ in $\mathbb{R}^{d}$ with radius $r$, where the distance is measured in $\lVert\cdot\rVert$. By $\lvert x\rvert$ we denote the euclidean norm of $x$ and by $\lVert x\rVert_{\ell^{\infty}}$ the maximum norm.

We endow $\mathbb{R}^{d}$ with the standard topology and for $A\subset\mathbb{R}^{d}$ we denote by $\overline{A}$ the closure of $A$ and by $\partial A$ the boundary of $A$. For the convex hull of $A$ we write $\operatorname{conv}A$. The diameter of a non-empty set $A\subset\mathbb{R}^{d}$ is always taken with respect to the euclidean distance, i.e. $\operatorname{diam}A:=\operatorname{diam}_{\lvert\cdot\rvert}A:=\sup_{x,y\in A}\lvert x-y\rvert$. If $A,B\subset\mathbb{R}^{d}$, then we write $A\subset\subset B$ if $\overline{A}$ is compact in $B$.

For $d_{1},d_{2}\in\mathbb{N}$ and a matrix $A\in\mathbb{R}^{d_{1},d_{2}}$ the number of nonzero entries of $A$ is counted by $\lVert\cdot\rVert_{\ell^{0}}$, i.e.

[TABLE]

If $d_{1},d_{2},d_{3}\in\mathbb{N}$ and $A\in\mathbb{R}^{d_{1},d_{2}},B\in\mathbb{R}^{d_{1},d_{3}}$, then we use the common block matrix notation and write for the horizontal concatenation of $A$ and $B$

[TABLE]

where the second notation is used if a stronger delineation between different blocks is necessary. A similar notation is used for the vertical concatenation of $A\in\mathbb{R}^{d_{1},d_{2}}$ and $B\in\mathbb{R}^{d_{3},d_{2}}$.

Of course, the same notation also applies to (block) vectors.

For a function $f:X\to\mathbb{R}$ we denote by $\mathrm{supp\leavevmode\nobreak\ }f$ the support of $f$. If $f:X\to Y$ and $g:Y\to Z$ are two functions, then we write $g\circ f:X\to Z$ for their composition.

We use the usual multiindex notation, i.e. for $\alpha\in\mathbb{N}_{0}^{d}$ we write $\lvert\alpha\rvert:=\alpha_{1}+\ldots+\alpha_{d}$ and $\alpha!:=\alpha_{1}!\cdot\ldots\cdot\alpha_{d}!$. Moreover, if $x\in\mathbb{R}^{d}$, then we have

[TABLE]

Let $\Omega\subset\mathbb{R}^{d}$ be open. For $n\in\mathbb{N}_{0}\cup\{\infty\}$, we denote by $C^{n}(\Omega)$ the set of $n$ times continuously differentiable functions on $\Omega$. The space of test functions is

[TABLE]

For $f\in C^{n}(\Omega)$ and $\alpha\in\mathbb{N}_{0}^{d}$ with $\lvert\alpha\rvert\leq n$ we write

[TABLE]

We say that a function $f:\Omega\to\mathbb{R}^{m}$ for $\Omega\subset\mathbb{R}^{d}$ is Lipschitz continuous if there is a constant $L>0$ such that

[TABLE]

for all $x,y\in\Omega$. If we want to emphasize the constant $L$, then we say that $f$ is $L$-Lipschitz.

If $X$ is a linear space and $\lVert\cdot\rVert_{1}$ and $\lVert\cdot\rVert_{2}$ are two norms on $X$, then we say that $\lVert\cdot\rVert_{1}$ and $\lVert\cdot\rVert_{2}$ are equivalent if there exist constants $C_{1},C_{2}>0$ such that

[TABLE]

For two normed linear spaces $X,Y$ we denote by $\mathcal{L}(X,Y)$ the set of bounded linear operators mapping $X$ to $Y$ and for $T\in\mathcal{L}(X,Y)$ the induced operator norm of $T$ is denoted by

[TABLE]

Let $a>0$, then we say for two functions $f:(0,a)\to[0,\infty)$ and $g:(0,a)\to[0,\infty)$ that $f(\varepsilon)$ is in $\mathcal{O}(g(\varepsilon))$ if there exists $0<\delta<a$ and $C>0$ such that $f(\varepsilon)\leq Cg(\varepsilon)$ for all $\varepsilon\in(0,\delta)$.

2 Neural networks

In this section, we introduce the notion of neural networks used in this paper. As in [61], we will consider a general type of feedforward architecture that also allows for connections of neurons in non-neighboring layers. It can be seen, though, that any function realized by such a network can also be realized by a network with a more restrictive feedforward architecture where only neurons from neighboring layers can be connected (Lemma 2.11). As in [47], we draw a distinction between the neural network and the function that the network realizes. This gives us the possibility to develop a network calculus in the spirit of [47, Chapter 2]. As in that paper we will introduce the notion of network concatenation and parallelization.

The following definition is similar to [47, Definition 2.1], where the difference is that we also allow connections between non-neighboring layers.

Definition 2.1**.**

Let $d,L\in\mathbb{N}$. A neural network $\Phi$ with input dimension $d$ and $L$ layers is a sequence of matrix-vector tuples

[TABLE]

where $N_{0}=d$ and $N_{1},\ldots,N_{L}\in\mathbb{N}$, and where each $A_{l}$ is an $N_{l}\times\sum_{k=0}^{l-1}N_{k}$ matrix, and $b_{l}\in\mathbb{R}^{N_{l}}$.

If $\Phi$ is a neural network as above, and if $\varrho:\mathbb{R}\to\mathbb{R}$ is arbitrary, then we define the associated realization of $\Phi$ with activation function $\varrho$ as the map $R_{\varrho}(\Phi):\mathbb{R}^{d}\to\mathbb{R}^{N_{L}}$ such that

[TABLE]

where $x_{L}$ results from the following scheme:

[TABLE]

where $\varrho$ acts componentwise, i.e., $\varrho(y)=[\varrho(y^{1}),\ldots,\varrho(y^{m})]$ for $y=[y^{1},\ldots,y^{m}]\in\mathbb{R}^{m}$. We sometimes write $A_{l}$ in block-matrix form as

[TABLE]

where $A_{l,x_{k}}$ is an $N_{l}\times N_{k}$ matrix for $k=0,\ldots,l-1$ and $l=1,\ldots,L$. Then

[TABLE]

We call $N(\Phi):=d+\sum_{j=1}^{L}N_{j}$ the number of neurons of the network $\Phi$, $L=L(\Phi)$ the number of layers, and finally $M(\Phi):=\sum_{j=1}^{L}(\lVert A_{j}\rVert_{\ell^{0}}+\lVert b_{j}\rVert_{\ell^{0}})$ denotes the number of nonzero entries of all $A_{l},b_{l}$ which we call the number of weights of $\Phi$. Moreover, we refer to $N_{L}$ as the dimension of the output layer of $\Phi$.

We will now define the class of neural networks where only connections between neighboring layers are allowed. Networks without skip connections are a special case of the networks of Definition 2.1. Since they are more frequently used in the literature, we coin such networks standard neural networks.

Definition 2.2**.**

If $\Phi=((A_{1},b_{1}),(A_{2},b_{2}),\dots,(A_{L},b_{L}))$ is a neural network as above and we have for

[TABLE]

that $A_{l,x_{i}}=0$ for $l=1,\ldots,L$ and $i=0,\ldots,l-2$, then we call $\Phi$ a standard neural network. The computation scheme then reduces to the following:

[TABLE]

In practice, before training a neural network, i.e. adjusting the weights of the network, one has to decide which network architecture to use. The following definition will clarify the notion of a network architecture.

Definition 2.3**.**

Let $d,L\in\mathbb{N}$. A neural network architecture ${\mathcal{A}}$ with input dimension $d$ and $L$ layers is a sequence of matrix-vector tuples

[TABLE]

where $N_{0}=d$ and $N_{1},\ldots,N_{L}\in\mathbb{N}$, and where each $A_{l}$ is an $N_{l}\times\sum_{k=0}^{l-1}N_{k}$ matrix, and $b_{l}$ a vector of length $N_{l}$ with elements in $\{0,1\}$. So a neural network architecture is simply a neural network with binary weights.

We call $N({\mathcal{A}}):=d+\sum_{j=1}^{L}N_{j}$ the number of neurons of the architecture ${\mathcal{A}}$, $L=L({\mathcal{A}})$ the number of layers, and finally $M({\mathcal{A}}):=\sum_{j=1}^{L}(\lVert A_{j}\rVert_{\ell^{0}}+\lVert b_{j}\rVert_{\ell^{0}})$ denotes the number of nonzero entries of all $A_{l},b_{l}$ which we call the number of weights of ${\mathcal{A}}$. Moreover, we refer to $N_{L}$ as the dimension of the output layer of ${\mathcal{A}}$.

We say that a neural network $\Phi=((A^{\prime}_{1},b^{\prime}_{1}),(A^{\prime}_{2},b^{\prime}_{2}),\dots,(A^{\prime}_{L},b^{\prime}_{L}))$ with input dimension $d$ and $L$ layers has architecture ${\mathcal{A}}$ if

1. (i)

$N_{l}(\Phi)=N_{l}$ for all $l=1,\ldots,L$ and 2. (ii)

$[A^{\prime}_{l}]_{i,j}\neq 0$ implies $[A_{l}]_{i,j}\neq 0$ for all $l=1,\ldots,L$ with $i=1,\ldots,N_{l}$ and $j=1,\ldots,\sum_{k=0}^{l-1}N_{k}$.

In the spirit of Definition 2.2 we say that ${\mathcal{A}}$ is a standard neural network architecture if all weights connecting neurons from non-neighboring layers are zero.

By fixing an ordering of the nonzero entries in $A_{l}$ for $l=1,\ldots,L$ we can uniquely identify a vector $w\in\mathbb{R}^{M({\mathcal{A}})}$ with a (standard) neural network that has (standard) architecture ${\mathcal{A}}$. We denote this (standard) neural network by ${\mathcal{A}}(w)$.

Note that in general an architecture of a neural network does not need to be unique.

Moreover, we define one of the in practice commonly used (see [37]) activation functions.

Definition 2.4**.**

The function

[TABLE]

is called ReLU (Rectified Linear Unit) activation function.

It can be shown, see [61, Proposition 1], that in terms of approximation and upper complexity bounds for networks with continuous and piecewise linear activation functions it suffices to focus on the ReLU.

2.1 Network concatenation

In this subsection we introduce the notion of concatenation of two networks which allows us to realize compositions of network realizations. In detail, if $\Phi^{1}$ and $\Phi^{2}$ are two networks, then we are interested in a network $\Phi^{1}\bullet\Phi^{2}$ such that $R_{\varrho}(\Phi^{1}\bullet\Phi^{2})(x)=R_{\varrho}(\Phi^{1})\circ R_{\varrho}(\Phi^{2})(x)$ for all $x\in\mathbb{R}^{d}$. We will start by constructing a network $\Phi^{1}\bullet\Phi^{2}$ for general activation functions $\varrho$. In the case where the activation function is the ReLU we use the fact that it is possible to find a network which realizes the identity function and construct a so called sparse concatenation which gives us control over the complexity of the resulting network.

Definition/Lemma 2.5**.**

Let $L_{1},L_{2}\in\mathbb{N}$ and let

[TABLE]

be two neural networks such that the input layer of $\Phi^{1}$ has the same dimension as the output layer of $\Phi^{2}$. Then, $\Phi^{1}\bullet\Phi^{2}$ denotes the following $L_{1}+L_{2}-1$ layer network:

[TABLE]

where

[TABLE]

for $l=1,\ldots,L_{1}$. We call $\Phi^{1}\bullet\Phi^{2}$ the concatenation of $\Phi^{1}$ and $\Phi^{2}$.

Then, $R_{\varrho}(\Phi^{1}\bullet\Phi^{2})=R_{\varrho}(\Phi^{1})\circ R_{\varrho}(\Phi^{2})$.

Proof.

This can be verified with a direct computation. ∎

Next, we adapt [47, Lemma 2.3] to the case where skip connections are allowed and define a network which realizes an identity function if the activation function is the ReLU.

Lemma 2.6**.**

Let $\varrho:\mathbb{R}\to\mathbb{R}$ be the ReLU and $d\in\mathbb{N}$. We define

[TABLE]

with

[TABLE]

Then $R_{\varrho}(\Phi^{\mathrm{Id}})=\mathrm{Id}_{\mathbb{R}^{d}}$.

Using the identity network $\Phi^{\mathrm{Id}}$ from the previous lemma we can now (as in [47, Definition 2.5]) define an alternative network concatenation which allows us to control the complexity of the resulting network for the case of the ReLU activation function.

Definition 2.7**.**

Let $\varrho:\mathbb{R}\to\mathbb{R}$ be the ReLU, let $L_{1},L_{2}\in\mathbb{N}$, and let

[TABLE]

be two neural networks such that the input dimension of $\Phi^{1}$ is equal to the output dimension of $\Phi^{2}$. Let $\Phi^{\mathrm{Id}}$ be as in Lemma 2.6.

Then the sparse concatenation of $\Phi^{1}$ and $\Phi^{2}$ is defined as

[TABLE]

Remark 2.8**.**

We have

[TABLE]

where

[TABLE]

for $l=1,\ldots,L_{1}$. Furthermore, it holds that $L(\Phi^{1}\odot\Phi^{2})=L_{1}+L_{2}$, and $M(\Phi^{1}\odot\Phi^{2})\leq 2M(\Phi^{1})+2M(\Phi^{2})$ and $N(\Phi^{1}\odot\Phi^{2})\leq 2N(\Phi^{1})+2N(\Phi^{2})$.

2.2 Network parallelization

In some cases it is useful to combine several networks into one larger network. In particular, if $\Phi^{1}$ and $\Phi^{2}$ are two networks, then we are looking for a network $P(\Phi^{1},\Phi^{2})$ such that $R_{\varrho}(P(\Phi^{1},\Phi^{2}))(x)=(R_{\varrho}(\Phi^{1})(x),R_{\varrho}(\Phi^{2})(x))$ for all $x\in\mathbb{R}^{d}$.

Definition/Lemma 2.9**.**

Let $L_{1},L_{2}\in\mathbb{N}$ and let

[TABLE]

be two neural networks with $d$-dimensional input. If $L_{1}\leq L_{2}$, then we define

[TABLE]

where

[TABLE]

and

[TABLE]

for $L_{1}\leq l<L_{2}$ and

[TABLE]

[TABLE]

A similar construction is used for the case $L_{1}>L_{2}$. Then $P(\Phi^{1},\Phi^{2})$ is a neural network with $d$-dimensional input and $\max\{L_{1},L_{2}\}$ layers, which we call the parallelization of $\Phi^{1}$ and $\Phi^{2}$. We have $M(P(\Phi^{1},\Phi^{2}))=M(\Phi^{1})+M(\Phi^{2})$, $N(P(\Phi^{1},\Phi^{2}))=N(\Phi^{1})+N(\Phi^{2})-d$ and $R_{\varrho}(P(\Phi^{1},\Phi^{2}))(x)=(R_{\varrho}(\Phi^{1})(x),R_{\varrho}(\Phi^{2})(x))$.

Remark 2.10**.**

The previous Definition 2.9 can be generalized to a parallelization of an arbitrary number of neural networks. Let $n\in\mathbb{N}$ and, for $i=1,\ldots,n$, let $\Phi^{i}$ be a neural network with $d$-dimensional input and $L_{i}$ layers. The network

[TABLE]

with $d$-dimensional input and $L=\max\{L_{1},\ldots,L_{n}\}$ layers is called the parallelization of $\Phi^{1},\Phi^{2},\ldots,\Phi^{n}$. We have

[TABLE]

and $R_{\varrho}(P(\Phi^{1},\ldots,\Phi^{n}))(x)=(R_{\varrho}(\Phi^{1})(x),\ldots,R_{\varrho}(\Phi^{n})(x))$.

2.3 Standard neural networks

In this subsection, we will show that if a function is a realization of a neural network, then it can also be realized by a standard neural network with similar complexity. The proof of this result is heavily based on the idea of the construction of an identity function with ReLU networks (Lemma 2.6).

Lemma 2.11**.**

Let $\varrho$ be the ReLU. Then there exist absolute constants $C_{1},C_{2}>0$ such that for any neural network $\Phi$ with $d$-dimensional input and $L=L(\Phi)$ layers, $N=N(\Phi)$ neurons and $M=M(\Phi)$ nonzero weights, there is a standard neural network $\Phi_{\text{st}}$ with $d$-dimensional input, and $L$ layers, at most $C_{1}LN$ neurons and $C_{2}\cdot(LN+M)$ nonzero weights such that

[TABLE]

for all $x\in\mathbb{R}^{d}$.

Proof.

If $L=1$, then there is nothing to show. For $L>1$ we start by defining the matrices

[TABLE]

and

[TABLE]

for $l=1,\ldots,L-1$ and

[TABLE]

Now we set

[TABLE]

and

[TABLE]

Then $\Phi_{\text{st}}$ is a standard neural network with $d$-dimensional input and $L$ layers and one can verify that $R_{\varrho}(\Phi)(x)=R_{\varrho}(\Phi_{\text{st}})(x)$ for all $x\in\mathbb{R}^{d}$. For the number of neurons in the $l$-th layer $N^{\text{st}}_{l}$ we have $N^{\text{st}}_{l}=2N_{0}+\sum_{k=1}^{l}N_{l}\leq 2N$. Hence, $N(\Phi_{\text{st}})\leq 2LN$. To estimate the number of weights, note that $\lVert A^{\text{st}}_{l}\rVert_{\ell^{0}}\leq 2N+2\lVert A_{l}\rVert_{\ell^{0}}$ and $\lVert b^{\text{st}}_{l}\rVert_{\ell^{0}}=\lVert b_{l}\rVert_{\ell^{0}}$, from which we conclude that $N(\Phi_{\text{st}})\leq 2(LN+M)$.

∎

In the next remark we collect some properties of the class of functions that can be realized by a neural network.

Remark 2.12**.**

If $\Phi$ is a neural network with $d$-dimensional input and $m$-dimensional output and $\varrho$ is the ReLU, then $R_{\varrho}(\Phi)$ is a Lipschitz-continuous, piecewise affine-linear function. This follows easily from Definition 2.2 and Lemma 2.11 which show that $R_{\varrho}(\Phi)$ can be expressed as the composition of Lipschitz-continuous, piecewise affine-linear transformations.

In [4, Theorem 2.1] also the converse was shown, i.e. every piecewise affine-linear function can be realized by a neural network with the ReLU as activation function.

3 Sobolev spaces

Spaces of functions that admit generalized derivatives fulfilling suitable integrability properties are a crucial concept in modern theory of partial differential equations (cf. e.g. [48, 20, 1]). In order to study properties of PDEs using functional analytic tools, a differential equation is reformulated via a differential operator mapping one function space to another. For a wide range of differential equations the appropriate spaces in this formulation are Sobolev spaces. A historical background of the development of Sobolev spaces in the context of PDEs can be found in [56, Lecture 1]. For a detailed treatment of the broad theory of Sobolev spaces we refer the reader to [1, 20, 11].

For this entire section, let $d\in\mathbb{N}$ and, if not stated otherwise, $\Omega\subset\mathbb{R}^{d}$ denote an open subset of $\mathbb{R}^{d}$. For $p\in[1,\infty]$, we denote by $L^{p}(\Omega)$ the standard Lebesgue spaces on $\Omega$. The following definition can be found in [11, Chapter 4].

Definition 3.1** (Sobolev space).**

Let $n\in\mathbb{N}_{0}$ and $1\leq p\leq\infty$. Then we define

[TABLE]

Furthermore, for $f\in{W^{n,p}(\Omega)}$ and $1\leq p<\infty$, we define the norm

[TABLE]

and

[TABLE]

We write $\nabla f:=(D^{1}f,\ldots,D^{d}f)$. Moreover, the space ${W^{n,p}(\Omega)}$ equipped with the norm $\lVert\cdot\rVert_{{W^{n,p}(\Omega)}}$ is a Banach space. For $f\in{W^{n,p}(\Omega)}$ we shall use the notation

[TABLE]

if the domain is clear from the context, or if we want to emphasize the variable $x$ that the function $f$ depends on, respectively. If $n=0$, then the Sobolev space is just a Lebesgue space, i.e. ${W^{0,p}(\Omega)}=L^{p}(\Omega)$.

Many results about function spaces defined on a domain $\Omega$ require $\Omega$ to fulfill certain regularity conditions. Different geometrical conditions and the resulting properties have been intensively studied and can for example be found in [1]. For our purposes it is enough to focus on the condition introduced in the next definition which can be found in [20, Appendix C.1].

Definition 3.2** (Lipschitz-domain).**

We say that a bounded and open set $\Omega\subset\mathbb{R}^{d}$ is a Lipschitz-domain if for each $x_{0}\in\partial\Omega$ there exists $r>0$ and a Lipschitz continuous function $g:\mathbb{R}^{d-1}\to\mathbb{R}$ such that

[TABLE]

after possibly relabeling and reorienting the coordinate axes.

In the sequel, we will only work with convex domains. Moreover, every open, bounded, and convex domain is a Lipschitz domain [23, Corollary 1.2.2.3].

3.1 Fractional Sobolev spaces

Fractional Sobolev spaces play an important role in the analysis of partial differential equations. In particular, they characterize the regularity of functions from Sobolev spaces defined on a domain $\Omega$ restricted to the boundary $\partial\Omega$ of the domain where the restriction is realized by a so-called trace operator (see e.g. [11]). Moreover, a detailed description of various areas of further application is listed in [12]. For a more in-depth analysis of these spaces the interested reader is referred to [38] and [56].

Next, we define fractional-order spaces in terms of an intrinsic norm.

Definition 3.3** (Sobolev-Slobodeckij spaces).**

We set for $0<s<1$ and $1\leq p\leq\infty$

[TABLE]

with

[TABLE]

and

[TABLE]

The space ${W^{s,p}(\Omega)}$ endowed with the norm $\lVert\cdot\rVert_{{W^{s,p}(\Omega)}}$ is a Banach space called Sobolev-Slobodeckij space.222For $p=\infty$ the space coincides with the space of bounded $s$-Hölder continuous functions. Precisely, each equivalence class contains a bounded $s$-Hölder continuous representative.

4 Approximations with deep ReLU neural networks in Sobolev type norms

In this section, we derive upper and lower complexity bounds for approximations of functions from certain Sobolev spaces with deep ReLU networks. Our result is a generalization of [61, Theorem 1] (upper bounds) and [61, Theorem 4 a)] (lower bounds) to the case where the approximation error is measured in a Sobolev-type norm.

From now on we assume that $\varrho:\mathbb{R}\to\mathbb{R}$ is the ReLU activation function (cf. Definition 2.4). We are interested in approximating functions in subsets of the Sobolev space ${W^{n,p}(\left(0,1\right)^{d})}$ with realizations of neural networks. For this we define the set:

[TABLE]

4.1 Upper complexity bounds

In [61, Theorem 1] it is shown that for $B=1$ and an arbitrary function $f\in\mathcal{F}_{n,d,\infty,B}$ a ReLU network can be constructed that realizes an approximation with $L^{\infty}$-error at most $\varepsilon$ using $\mathcal{O}(\log_{2}(\nicefrac{{1}}{{\varepsilon}}))$ layers and $\mathcal{O}(\varepsilon^{-d/n}\log_{2}(\nicefrac{{1}}{{\varepsilon}}))$ nonzero weights and neurons.

We will show that the approximation can also be performed with respect to a continuous scale of higher order Sobolev-type norms for arbitrary $1\leq p\leq\infty$ and additionally, that there is a trade-off between the regularity used in the norm in which the approximation error is measured and the regularity used in the bounds. In particular, we will show the following theorem:

Theorem 4.1**.**

Let $d\in\mathbb{N}$, $n\in\mathbb{N}_{\geq 2}$, $1\leq p\leq\infty$, $B>0$, and $0\leq s\leq 1$. Then, there exists a constant $c=c(d,n,p,B,s)>0$ with the following properties:

For any $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$, there is a neural network architecture ${\mathcal{A}}_{\varepsilon}={\mathcal{A}}_{\varepsilon}(d,n,p,B,s,\varepsilon)$ with $d$-dimensional input and one-dimensional output such that for any $f\in\mathcal{F}_{n,d,p,B}$ there is a neural network $\Phi_{\varepsilon}^{f}$ that has architecture ${\mathcal{A}}_{\varepsilon}$ such that

[TABLE]

and

1. (i)

$L({\mathcal{A}}_{\varepsilon})\leq c\cdot\log_{2}(\varepsilon^{-n/(n-s)})$; 2. (ii)

$M({\mathcal{A}}_{\varepsilon})\leq c\cdot\varepsilon^{-d/(n-s)}\cdot\log_{2}(\varepsilon^{-n/(n-s)})$; 3. (iii)

$N({\mathcal{A}}_{\varepsilon})\leq c\cdot\varepsilon^{-d/(n-s)}\cdot\log_{2}(\varepsilon^{-n/(n-s)})$.

Clearly, if $s=0$ and $p=\infty$, then Theorem 4.1 coincides with the theorem shown by Yarotsky in [61]. Note that if $\Phi$ is a neural network and $\varrho$ is the ReLU, then the restriction of $R_{\varrho}(\Phi)$ to $\left(0,1\right)^{d}$ is bounded and Lipschitz continuous (cf. Remark 2.12) and hence in view of [10, Chapter 1.3] an element of the Sobolev space ${W^{1,\infty}(\left(0,1\right)^{d})}$. Therefore, the expressions in the previous theorem are well-defined.

Remark**.**

Theorem 4.1 holds for all activation functions that are in some sense similar to the ReLU. In particular, it follows directly from [61, Proposition 1] that the statement holds for any continuous piecewise linear activation function $\hat{\varrho}$ with $M$ breakpoints, where $1\leq M<\infty$.

As a consequence of Theorem 4.1 and Lemma 2.11, we can easily derive a similar statement for standard neural networks.

Corollary 4.2**.**

Let $d\in\mathbb{N}$, $n\in\mathbb{N}_{\geq 2}$, $1\leq p\leq\infty$, $B>0$, and $0\leq s\leq 1$. Then, there exists a constant $c=c(d,n,p,B,s)>0$ with the following properties:

For any $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$, there is a standard neural network architecture ${\mathcal{A}}_{\text{st},\varepsilon}={\mathcal{A}}_{\text{st},\varepsilon}(d,n,p,B,s,\varepsilon)$ with $d$-dimensional input and one-dimensional output such that for any $f\in\mathcal{F}_{n,d,p,B}$ there is a standard neural network $\Phi_{\text{st},\varepsilon}$ that has architecture ${\mathcal{A}}_{\text{st},\varepsilon}$ such that

[TABLE]

and

1. (i)

$L({\mathcal{A}}_{\text{st},\varepsilon})\leq c\cdot\log_{2}(\varepsilon^{-n/(n-s)})$; 2. (ii)

$M({\mathcal{A}}_{\text{st},\varepsilon})\leq c\cdot\varepsilon^{-d/(n-s)}\cdot\log_{2}^{2}(\varepsilon^{-n/(n-s)})$; 3. (iii)

$N({\mathcal{A}}_{\text{st},\varepsilon})\leq c\cdot\varepsilon^{-d/(n-s)}\cdot\log_{2}^{2}(\varepsilon^{-n/(n-s)})$.

4.2 Lower complexity bounds

In this subsection, we show that the same regularity-complexity trade-off that can be observed for upper complexity bounds can be shown for lower bounds. In Theorem 4 a) in [61] Yarotsky proves that a network architecture capable of approximating any function $f\in\mathcal{F}_{n,d,\infty,1}$ up to a $L^{\infty}$-error $\varepsilon$ has at least $c\cdot\varepsilon^{-d/(2n)}$ weights. Here we consider approximations in $W^{1,\infty}$ norm. The following theorem combines the result from [61] with our result.

Theorem 4.3**.**

Let $d\in\mathbb{N}$, $n\in\mathbb{N}_{\geq 2}$, $B>0$ and $k\in\{0,1\}$. Then, there is a constant $c=c(d,n,B,k)>0$ with the following property:

If $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$ and ${\mathcal{A}}_{\varepsilon}={\mathcal{A}}_{\varepsilon}(d,n,B,k,\varepsilon)$ is an architecture such that for any $f\in\mathcal{F}_{n,d,\infty,B}$ there is a neural network $\Phi_{\varepsilon}^{f}$ that has architecture ${\mathcal{A}}_{\varepsilon}$ and

[TABLE]

then ${\mathcal{A}}_{\varepsilon}$ has at least $M({\mathcal{A}}_{\varepsilon})\geq c\varepsilon^{\nicefrac{{-d}}{{2(n-k)}}}$ weights.

In case $k=0$, Theorem 4.3 coincides with [61, Theorem 4 a)].

Remark**.**

Note that the case of standard neural network architectures is included in the theorem above and, thus, the same lower bounds hold true for this case.

Moreover, the proof can easily be adapted to piecewise polynomial activation functions (not necessarily continuous) with a finite number of breakpoints.

5 Discussion and future work

The motivation behind deriving theoretical results for deep neural networks is to improve our understanding of their empirical success in a broad area of applications. In this section, we briefly describe the practical relevance of our result, but mostly focus on aspects where our theoretical framework differs from empirical observations or fails to incorporate computational limitations met in implementations. Furthermore, we will discuss possible extensions of our results.

• •

Practical relevance. Our results give upper and lower bounds for the complexity of neural network architectures used in Sobolev training (see Section 1.3 and [16]). Furthermore, we anticipate our results to lead to complexity bounds in the theoretical foundation of deep learning-based approaches for the numerical solution of PDEs. We leave an in-depth analysis of this topic for future research.

• •

Curse of dimension. One of the main reasons for the current interest in deep neural networks is their outstanding performance in high-dimensional problems. As an example, consider the popular ILSVR333Typically abbreviated with ILSRC which stands for ImageNet Large Scale Visual Recognition Challenge. challenge (see [49]), an image recognition task on the ImageNet database (see [18]) containing variable-resolution images which are typically downsampled before training to yield a roughly $(240\times 240\times 3)$-dimensional input space (cf. e.g. [35, 64, 53]).

Our asymptotic upper complexity bounds for the number of weights and neurons are in $\mathcal{O}(\varepsilon^{-d/(n-s)})$, and thus depend strongly on the dimension $d$ of the input space. Moreover, the constants in the upper bounds for the number of layers, weights, and neurons also increase exponentially with increasing $d$. Even if there is still a gap between our upper and lower bounds for the weights of a network architecture, Theorem 4.3 shows that one cannot expect to circumvent the curse of dimension in the setting considered in this paper.

Several ways have been proposed so far to tackle this common problem. One idea is to think of the data as residing on or near a manifold $\mathcal{M}$ embedded in $\mathbb{R}^{d}$ with dimension $d_{\mathcal{M}}\ll d$ (see [7]). Considering, for example, image classification problems this idea seems to be rather intuitive since most of the elements in $\mathbb{R}^{240\times 240\times 3}$ are not perceived as images. A similar approach is to narrow down the approximated function space by incorporating invariances (see [39]). If, for example, the approximated function $f$ maps images to labels, then it makes sense to assume that $f$ is translation and rotation invariant. The additional structure of the function space can then be exploited in the approximation (see [47, Section 5]). In particular, the last approach could also be suitable to tackle the curse of dimension in our result.

• •

The power of depth. For $s=0$, the approximations obtained in Theorem 4.1 and [61, Theorem 1] coincide. In this case, Yarotsky showed in [61] that the constructed unbounded depth approximations for functions in ${W^{n,\infty}(\left(0,1\right)^{d})}$ (with $n>2$) are asymptotically more efficient in the number of weights and neurons than approximations with fixed length $L$ if

[TABLE]

As a consequence, to be more efficient than a shallow network, i.e. a network with depth $L=2$, one needs $n>2d$ regularity. Even if this result does not completely explain the success of deep networks over shallow ones, since $d$ is typically very large, it would be interesting to obtain similar results for higher-order Sobolev norms.

• •

Unbounded complexity of weights. When training a neural network on a computer, the weights have to be stored in memory. In practice, storing weights up to an arbitrary precision or even unbounded weights (in absolute value) is infeasible. In the construction of the neural network $R_{\varrho}(\Phi^{f}_{\varepsilon})$ that approximates a function $f$ up to an $W^{s,\infty}$-error $\varepsilon$ and realizes the upper complexity bound from Theorem 4.1 we used weights $W_{\varepsilon}$ with $\lvert W_{\varepsilon}\rvert\to\infty$ as $\varepsilon\downarrow 0$ (see construction of $\Phi_{m,l}$ in the proof of Lemma C.3 (v)).

In [47, 47, 24], neural networks are restricted to possess quantized weights (see [47, Definition 2.9]) which controls the complexity of the weights depending on the approximation error $\varepsilon$. It would be interesting to extend Theorem 4.1 to networks with quantized weights and to analyze how coarse a quantization is possible to maintain the associated upper bounds.

Acknowledgements

I.G. would like to thank Mones Raslan for fruitful discussions on the topic and acknowledges support from the Research Training Group ”Differential Equation- and Data-driven Models in Life Sciences and Fluid Dynamics: An Interdisciplinary Research Training Group (DAEDALUS)” (GRK 2433) funded by the German Research Foundation (DFG). G. K. acknowledges partial support by the Bundesministerium für Bildung und Forschung (BMBF) through the Berliner Zentrum for Machine Learning (BZML), Project AP4, by the Deutsche Forschungsgemeinschaft (DFG) through grants CRC 1114 ”Scaling Cascades in Complex Systems”, Project B07, CRC/TR 109 ”Discretization in Geometry and Dynamics’, Projects C02 and C03, RTG DAEDALUS (RTG 2433), Projects P1 and P3, RTG BIOQIC (RTG 2260), Projects P4 and P9, and SPP 1798 ”Compressed Sensing in Information Processing”, Coordination Project and Project Massive MIMO-I/II, by the Berlin Mathematics Research Center MATH+ , Projects EF1-1 and EF1-4, and by the Einstein Foundation Berlin. P.P. is supported by a DFG Research Fellowship ”Shearlet-based energy functionals for anisotropic phase-field methods”.

Appendix A Interpolation spaces

In this section, we recall some essential notations and results from interpolation theory in Banach spaces. In particular, we present the $K$-method of real interpolation.

For two Banach spaces $B_{0}$ and $B_{1}$ interpolation techniques allow the definition and study of a family of spaces that in some sense bridge the gap between $B_{0}$ and $B_{1}$. This rather abstract concept will be applied to Sobolev spaces in Section 3.1 to derive spaces of fractional regularity. Interpolation theory is a valuable tool in harmonic analysis and the study of partial differential equations. A small summary about the history of the development of the theory of interpolation spaces can be found in [56, Lecture 21]. Interested readers are recommended to read the monographs [38, 8, 56] and [57] for a detailed treatment in the context of partial differential equations.

The basic notions of functional analysis are assumed to be known and we refer to [62] for an introduction.

If $B_{0},B_{1}$ are two Banach spaces such that $B_{1}$ is continuously embedded in $B_{0}$, then we write $B_{1}\subset B_{0}$ and call $(B_{0},B_{1})$ an interpolation couple. One can also consider more general pairs of Banach spaces but this setting is sufficiently general for our purposes.

Definition A.1**.**

Let $(B_{0},B_{1})$ be an interpolation couple. For every $u\in B_{1}$ and $t>0$ we define

[TABLE]

The first term measures how well $u$ can be approximated by elements from the smaller space $B_{1}$ in $\lVert\cdot\rVert_{B_{0}}$ and the second term is a penalty term weighted by $t$.

Now we can define interpolation spaces.

Definition A.2**.**

Let $(B_{0},B_{1})$ be an interpolation couple. Let $0<\theta<1$ and $1\leq p\leq\infty$. We set

[TABLE]

Moreover, we define the set

[TABLE]

The norm $u\mapsto\lVert u\rVert_{\left(B_{0},B_{1}\right)_{\theta,p}}$ turns $\left(B_{0},B_{1}\right)_{\theta,p}$ into a Banach space (cf. [38, Proposition 1.5]) which is called a real interpolation space.

The next lemma shows that in the sense of a continuous embedding the spaces $B_{\theta,p}$ form a family of nested Banach spaces.

Lemma A.3**.**

Let $1\leq p\leq\infty$, then the following holds:

1. (i)

If $(B_{0},B_{1})$ is an interpolation couple and $0<\theta_{1}<\theta_{2}<1$, then

[TABLE] 2. (ii)

if $B$ is a Banach space and $0<\theta<1$, then $\left(B,B\right)_{\theta,p}=B$.

Proof.

For (i) see [38, p. 2] and for (ii) [38, Proposition 1.4]. ∎

One of the most important theorems in interpolation theory shows how the norm of an operator defined on interpolation couples relates to the operator norm with respect to the corresponding interpolation spaces. The theorem can be found in [38, Theorem 1.6].

Theorem A.4**.**

Let $(A_{0},A_{1})$ and $(B_{0},B_{1})$ be two interpolation couples. If $T\in\mathcal{L}(A_{0},B_{0})\cap\mathcal{L}(A_{1},B_{1})$, then $T\in\mathcal{L}(A_{\theta,p},B_{\theta,p})$ for every $0<\theta<1$ and $1\leq p\leq\infty$. Furthermore,

[TABLE]

As a corollary from the previous theorem the following useful result can be obtained (see [38, Corollary 1.7]).

Corollary A.5**.**

Let $(B_{0},B_{1})$ be an interpolation couple. Moreover, let $0<\theta<1$ and $1\leq p\leq\infty$, then there is a constant $c=c(\theta,p)>0$ such that for all $u\in B_{1}$ we have

[TABLE]

For the case $p=\infty$ we have $c(\theta,\infty)=1$.

Appendix B Sobolev spaces

In this section, we give a brief introduction to Sobolev spaces and present related concepts which are important for the proof of Theorem 4.1. In detail, we show estimates for compositions and products of functions from certain Sobolev spaces in Subsection B.1 and B.2, respectively. In Subsection B.3, we introduce a generalization of Taylor polynomials which is one of the key tools in the proof of Theorem 4.1. In the last subsection, we apply Banach space interpolation theory to (integer) Sobolev spaces to define fractional Sobolev spaces. Again, let for this entire section, $d\in\mathbb{N}$ and, if not stated otherwise, $\Omega\subset\mathbb{R}^{d}$ denote an open subset of $\mathbb{R}^{d}$.

As a generalization of Definition 3.1, we now introduce Sobolev spaces of vector-valued functions (see [48, Chapter 1.2.3]).

Definition B.1**.**

We set for $n\in\mathbb{N}_{0}$, $m\in\mathbb{N}$ and $1\leq p\leq\infty$

[TABLE]

with

[TABLE]

and

[TABLE]

These spaces are again Banach spaces. To simplify the exposition, we introduce notation for a family of semi-norms.

Definition B.2** (Sobolev semi-norm).**

For $n,k\in\mathbb{N}_{0}$ with $k\leq n$, $m\in\mathbb{N}$ and $1\leq p\leq\infty$ we define for $f\in{W^{n,p}(\Omega;\,\mathbb{R}^{m})}$ the Sobolev semi-norm

[TABLE]

and

[TABLE]

For $m=1$ we only write $\lvert\cdot\rvert_{{W^{k,p}(\Omega)}}$. Note that $\lvert\cdot\rvert_{{W^{0,p}(\Omega)}}$ coincides with $\lVert\cdot\rVert_{L^{p}(\Omega)}$.

Rademacher’s Theorem states that a Lipschitz continuous function $f:\Omega\to\mathbb{R}$ is differentiable (in the classical sense) almost everywhere in $\Omega$ (see e.g. [20, Theorem 5.8.6]). The following theorem yields an alternative characterization for the space $W^{1,\infty}$ using Lipschitz continuous functions and connects the notion of the classical and the weak derivative.

Theorem B.3**.**

Let $\Omega\subset\mathbb{R}^{d}$ be open and convex. If $f\in W^{1,\infty}(\Omega)$, then $f$ is $\lVert\;\lvert\nabla f(x)\rvert\;\rVert_{L^{\infty}(\Omega;dx)}$-Lipschitz. Conversely, if $f$ is bounded and $L$-Lipschitz on $\Omega$, then $f\in W^{1,\infty}(\Omega)$, the weak gradient of $f$ agrees almost everywhere with the classical gradient, and

[TABLE]

A proof can be found in [29, Theorem 4.1] together with [29, Remark 4.2].

To use the bounds for the gradient obtained in the previous theorem in the Sobolev semi-norm setting from Definition B.2, the following observation turns out to be useful. If $f\in W^{1,\infty}(\Omega)$, then

[TABLE]

Moreover, it is not hard to see that if $\Omega\subset\mathbb{R}^{d}$ open and bounded, then ${W^{n,\infty}(\Omega)}\subset{W^{n,p}(\Omega)}$ for $1\leq p<\infty$, and $n\in\mathbb{N}_{0}$.

B.1 Composition estimate

In this subsection, we derive an estimate for the semi-norm $\lvert\cdot\rvert_{W^{1,\infty}}$ of the composition $g\circ f$ of two functions $f\in{W^{1,\infty}(\Omega;\,\mathbb{R}^{m})}$ and $g\in{W^{1,\infty}(\mathbb{R}^{m})}$. Since in general this is not a well-defined notion we need to clarify what $g\circ f$ denotes in this case. To demonstrate the problem let $f$ and $g$ be as above and recall that $f$ and $g$ are equivalence classes of functions. Let now $\hat{f}$ be a representative for the equivalence class $f$ and $\hat{g}_{1},\hat{g}_{2}$ be two representatives for $g$. Then $\hat{g}_{1}=\hat{g}_{2}$ almost everywhere but in general we do not have $\hat{g}_{1}\circ\hat{f}=\hat{g}_{2}\circ\hat{f}$ almost everywhere. To circumvent this problem we denote by $g\circ f$ the equivalence class of $\hat{g}\circ\hat{f}$ where $\hat{g}$ and $\hat{f}$ are Lipschitz continuous representatives of $g$ and $f$, respectively.

The next lemma recalls a well known fact about the composition of Lipschitz functions and is not hard to see.

Lemma B.4**.**

Let $d,m\in\mathbb{N}$ and $\Omega_{1}\subset\mathbb{R}^{d}$, $\Omega_{2}\subset\mathbb{R}^{m}$. Let $f:\Omega_{1}\to\Omega_{2}$ be a $L_{1}$-Lipschitz function and $g:\Omega_{2}\to\mathbb{R}$ be a $L_{2}$-Lipschitz function for some $L_{1},L_{2}>0$. Then the composition $g\circ f:\Omega_{1}\to\mathbb{R}$ is a $(L_{1}\cdot L_{2})$-Lipschitz function.

The following corollary establishes a chain rule estimate for $W^{1,\infty}$.

Corollary B.5**.**

Let $d,m\in\mathbb{N}$ and $\Omega_{1}\subset\mathbb{R}^{d}$, $\Omega_{2}\subset\mathbb{R}^{m}$ both be open, bounded, and convex. Then, there is a constant $C=C(d,m)>0$ with the following property:

If $f\in{W^{1,\infty}(\Omega_{1};\,\mathbb{R}^{m})}$ and $g\in{W^{1,\infty}(\Omega_{2})}$ are Lipschitz continuous representatives such that $\operatorname{ran}f\subset\Omega_{2}$, then for the composition $g\circ f$ it holds that $g\circ f\in{W^{1,\infty}(\Omega_{1})}$ and we have

[TABLE]

and

[TABLE]

Proof.

We set for $j=1,\ldots,m$

[TABLE]

It then follows from Theorem B.3 that $f_{j}$ is $L_{j}$-Lipschitz and hence, $f$ is $\lvert L_{f}\rvert$-Lipschitz. In the same manner we define

[TABLE]

and have that $g$ is $L_{g}$-Lipschitz. With Lemma B.4 we conclude that also the composition $g\circ f$ is $(\lvert L_{f}\rvert\cdot L_{g})$-Lipschitz. Since $g$ is bounded and hence also $g\circ f$ applying Theorem B.3 again yields that $g\circ f\in{W^{1,\infty}(\Omega_{1})}$ and furthermore

[TABLE]

We get

[TABLE]

where we used Equation (B.1) in the first step and the previous estimate (B.2) in the second. The third step is due to estimating the $\ell^{2}$ norm with the $\ell^{\infty}$ norm on $\mathbb{R}^{m}$, and applying (B.1) to the definition of $L_{g}$. The last step uses Equation (B.1) again. This shows the first estimate. The second estimate is now an easy consequence of the first. ∎

B.2 Product estimate

In the following lemma, we show an estimate for the semi-norm of a product of weakly differentiable functions.

Lemma B.6**.**

Let $f\in{W^{1,\infty}(\Omega)}$ and $g\in{W^{1,p}(\Omega)}$ with $1\leq p\leq\infty$, then $fg\in{W^{1,p}(\Omega)}$ and there exists a constant $C=C(d,p)>0$ such that

[TABLE]

For $p=\infty$, we have $C=1$.

Proof.

Clearly, $fg,fD^{i}g+gD^{i}f\in L^{1}_{\text{loc}}(\Omega)$ so that the product formula in [21, Chapter 7.3] yields that for the weak derivatives of $fg$ it holds

[TABLE]

for $i=1,2,\ldots,d$. Thus, we have

[TABLE]

Note that we have

[TABLE]

and

[TABLE]

Finally, we get

[TABLE]

where $C=C(d,p)>0$ is a constant. Furthermore, it is clear that also $fg\in L^{p}(\Omega)$. ∎

B.3 Averaged Taylor polynomial

In this subsection, we develop a polynomial approximation in the spirit of Taylor polynomials but appropriate for Sobolev spaces. A polynomial approximation $P_{f}$ of a function $f\in\mathcal{F}_{n,d,p,B}$ is the first step towards an approximation of $f$ realized by a neural network in the proof of Theorem 4.1.

A reference for this entire subsection is [10, Chapter 4.1].

Definition B.7** (averaged Taylor polynomial).**

Let $n\in\mathbb{N}$, $1\leq p\leq\infty$ and $f\in{W^{n-1,p}(\Omega)}$, and let $x_{0}\in\Omega$, $r>0$ such that for the ball $B:=B_{r,\lvert\cdot\rvert}(x_{0})$ it holds that $B\subset\subset\Omega$. The corresponding Taylor polynomial of order $n$ of $f$ averaged over $B$ is defined for $x\in\Omega$ as

[TABLE]

where

[TABLE]

and $\phi$ is an arbitrary cut-off function supported in $\overline{B}$, i.e.

[TABLE]

A cut-off function as used in the previous definition always exists. A possible choice is

[TABLE]

normalized by $\int_{\mathbb{R}^{d}}\psi(x)dx$. Next, we derive some properties of the averaged Taylor polynomial.

Remark B.8**.**

From the linearity of the weak derivative we can easily conclude that the averaged Taylor polynomial is linear in $f$.

Recall that the averaged Taylor polynomial is defined via an integral and some cut-off function (cf. (B.3)) that perform an averaging of a polynomial expression (cf. (B.4)). Additionally, the following lemma shows that the averaged Taylor polynomial of order $n$ is indeed a polynomial of degree less than $n$ in $x$.

Lemma B.9**.**

Let $n\in\mathbb{N}$, $1\leq p\leq\infty$ and $f\in{W^{n-1,p}(\Omega)}$, and let $x_{0}\in\Omega$, $r>0,R\geq 1$ such that for the ball $B:=B_{r,\lvert\cdot\rvert}(x_{0})$ it holds that $B\subset\subset\Omega$ and $B\subset B_{R,\lVert\cdot\rVert_{\ell^{\infty}}}(0)$. Then the Taylor polynomial of order $n$ of $f$ averaged over $B$ can be written as

[TABLE]

for $x\in\Omega$.

Moreover, there exists a constant $c=c(n,d,R)>0$ such that the coefficients $c_{\alpha}$ are bounded with $\lvert c_{\alpha}\rvert\leq cr^{-d/p}\lVert f\rVert_{{W^{n-1,p}(\Omega)}}$ for all $\alpha$ with $\lvert\alpha\rvert\leq n-1$.

Proof.

The first part of the proof of this lemma follows closely the chain of arguments in [10, Equations (4.1.5) - (4.1.8)]. We write for $\alpha\in\mathbb{N}_{0}^{d}$

[TABLE]

where $a_{(\gamma,\beta)}\in\mathbb{R}$ are suitable constants with

[TABLE]

in multi-index notation. Then, combining Equation (B.3) and (B.4) yields

[TABLE]

For the second part, note that

[TABLE]

where we used $B\subset B_{R,\lVert\cdot\rVert_{\ell^{\infty}}}(0)$ and the Hölder’s inequality with $1/q=1-1/p$. Next, since $\phi\in L^{1}(B)\cap L^{\infty}(B)$ and $\lVert\phi\rVert_{L^{1}}=1$ we have (see [2, Chapter X.4 Exercise 4])

[TABLE]

Combining the last estimate with Equation (B.7) yields

[TABLE]

where the second step follows from $\lVert\phi\rVert_{L^{\infty}}\leq cr^{-d}$ for some constant $c=c(d)>0$ (see [10, Section 4.1]). To estimate the absolute value of the coefficients $c_{\gamma}$ (defined in Equation B.6), we have

[TABLE]

Here, the second step used Equation (B.8) together with Equation (B.5), and $c^{\prime}=c^{\prime}(n,d,R)>0$ is a constant. ∎

The next step is to derive approximation properties of the averaged Taylor polynomial. To this end, recall that for the (standard) Taylor expansion of some function $f$ defined on a domain $\Omega$ in $x_{0}$ to yield an approximation at some point $x_{0}+h$ the whole path $x_{0}+th$ for $0\leq t\leq 1$ has to be contained in $\Omega$ (see [40, Theorem 6.8.10]). In case of the averaged Taylor polynomial the expansion point $x_{0}$ is replaced by a ball $B$ and we require that the path between each $x_{0}\in B$ and each $x\in\Omega$ is contained in $\Omega$. This geometrical condition is made precise in the following definition.

Definition B.10**.**

Let $\Omega,B\subset\mathbb{R}^{d}$. Then $\Omega$ is called star-shaped with respect to $B$ if,

[TABLE]

The next definition introduces a geometric notion which becomes important when given a family of subdivisions $\mathcal{T}^{h}$, $0<h\leq 1$ of a domain $\Omega$ which becomes finer for smaller $h$. One typically needs to control the nondegeneracy of $(\mathcal{T}^{h})_{h}$ which can be done e.g. with a uniformly bounded chunkiness parameter.

Definition B.11**.**

Let $\Omega\subset\mathbb{R}^{d}$ be bounded. We define the set

[TABLE]

If $\mathcal{R}\neq\varnothing$, then we define

[TABLE]

the chunkiness parameter of $\Omega$.

To emphasize the dependence on the set $\Omega$, we will occasionally write $r^{\star}_{\text{max}}(\Omega)$ and $\gamma(\Omega)$.

Finally, the next lemma shows approximation properties of the averaged Taylor polynomial.

Lemma B.12** (Bramble-Hilbert).**

Let $\Omega\subset\mathbb{R}^{d}$ be open and bounded, $x_{0}\in\Omega$ and $r>0$ such that $\Omega$ is star-shaped with respect to $B:=B_{r,\lvert\cdot\rvert}(x_{0})$, and $r>(1/2)r^{\star}_{\text{max}}$. Moreover, let $n\in\mathbb{N}$, $1\leq p\leq\infty$ and denote by $\gamma$ the chunkiness parameter of $\Omega$. Then there exists a constant $C=C(n,d,\gamma)>0$ such that for all $f\in{W^{n,p}(\Omega)}$

[TABLE]

where $Q^{n}f$ denotes the Taylor polynomial of order $n$ of $f$ averaged over $B$ and $h=\operatorname{diam}(\Omega)$.

A proof can be found in [10, Lemma 4.3.8].

B.4 Fractional Sobolev spaces

In this subsection, we derive a generalization of Sobolev spaces characterized by fractional-order derivatives using two different approaches. First, we interpolate integer-valued Sobolev spaces, and secondly, we define an intrinsic norm. Both approaches are then shown to be equivalent under some regularity condition on the domain $\Omega$.

We start by defining fractional-order Sobolev spaces $W^{s,p}$ for $0<s<1$ via Banach space interpolation (see Section A).

Definition B.13**.**

Let $0<s<1$ and $1\leq p\leq\infty$, then we set

[TABLE]

The next theorem shows that, given suitable regularity conditions of the domain $\Omega$, Definition B.13 and Definition 3.3 yield the same spaces with equivalent norms. The theorem can be found in [10, Theorem 14.2.3] for $1\leq p<\infty$. The case $p=\infty$ is not included in [10, Theorem 14.2.3] but can be shown with the same technique.

Theorem B.14**.**

Let $\Omega\subset\mathbb{R}^{d}$ be a Lipschitz-domain. Then we have for $0<s<1$ and $1\leq p\leq\infty$ that

[TABLE]

with equivalence of the respective norms.

This motivates the following remark.

Remark**.**

We make use of the norm equivalence by only using the $W^{s,p}$ norm in the following section and transferring results obtained by interpolation theory for the $\widetilde{W}^{s,p}$ norm to the $W^{s,p}$ norm without mentioning it again.

Appendix C Upper bounds for approximations

The strategy of the proof of Theorem 4.1 follows the idea of the proof of Theorem 1 in [61]: A polynomial approximation $P_{f}$ of a function $f\in\mathcal{F}_{n,d,p,B}$ is approximated by a function realized by a neural network $\Phi_{P_{f}}$. As in [61], we start by constructing a neural network that approximates the square function on the interval $\left(0,1\right)$ up to an approximation error at most $\varepsilon$. In our result the error is measured in the $W^{1,\infty}$ norm (Proposition C.1). This result is then used to obtain an approximation of a multiplication operator (Proposition C.2). Next, we derive a partition of unity (Lemma C.3) as a product of univariate functions that can be realized by neural networks. Using this construction we then build a localized Taylor approximation for a function $f\in\mathcal{F}_{n,d,p,B}$ in the $L^{p}$ and $W^{1,p}$ norm. An interpolation argument shows that same construction can be used for an approximation in the $W^{s,p}$ norm, where $0\leq s\leq 1$ (Lemma C.4). In the next step, we lay the foundation for approximating products of the partition of unity and monomials, i.e. localized monomials, with ReLU networks (Lemma C.5) in the $W^{k,\infty}$ norm for $k=0,1$. Via an embedding this result this result is then used to approximate the localized polynomials in $W^{k,p}$ with ReLU networks (Lemma C.6).

All Sobolev spaces encountered in the following proofs are defined on open, bounded, and convex domains which are, in particular, Lipschitz-domains (see [23, Corollary 1.2.2.3]).

The results presented in the following two propositions can be found in a similar way in [51, Proposition 3.1]. However, since our results contain some minor extensions we decided to give the proof here for the sake of completeness. In detail, Proposition C.2 (ii) and (iii) are not shown in [51, Proposition 3.1].

In [61, Proposition 2], a network is constructed that approximates the square function $x\to x^{2}$ on the interval $\left(0,1\right)$ in the $L^{\infty}$ norm. Interestingly, the same construction can be used when measuring the approximation error in the $W^{1,\infty}$ norm. In particular, the depth and the number of weights and neurons of the network do not grow asymptotically faster to satisfy the approximation accuracy with respect to this stronger norm.

Proposition C.1**.**

There exist constants $c_{1},c_{2},c_{3},c_{4}>0$, such that for all $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$ there is a neural network $\Phi^{\text{sq}}_{\varepsilon}$ with at most $c_{1}\cdot\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ nonzero weights, at most $c_{2}\cdot\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ layers, at most $c_{3}\cdot\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ neurons, and with one-dimensional input and output such that

[TABLE]

and $R_{\varrho}(\Phi^{\text{sq}}_{\varepsilon})(0)=0$. Furthermore, it holds that

[TABLE]

Proof.

In the proof of [61, Proposition 2] it is shown that there exist constants $c_{1}$, $c_{2}$, $c_{3}>0$, such that for each $m\in\mathbb{N}$ there is a neural network $\Phi_{m}$ with at most $c_{1}\cdot m$ nonzero weights, at most $c_{2}\cdot m$ layers, at most $c_{3}\cdot m$ neurons the realization of which is a piecewise linear interpolation of $x\mapsto x^{2}$ on $\left(0,1\right)$. In detail, it is shown there, that the network $\Phi_{m}$ satisfies for all $k\in\{0,\ldots,2^{m}-1\}$ and all $x\in\left[\frac{k}{2^{m}},\frac{k+1}{2^{m}}\right]$

[TABLE]

Thus, $R_{\varrho}(\Phi_{m})$ is a piecewise linear interpolant of $f$ with $2^{m}+1$ uniformly distributed breakpoints $\frac{k}{2^{m}},k=0,\ldots,2^{m}$. In particular, $R_{\varrho}(\Phi_{m})(0)=0$. Furthermore, it is shown in the proof of [61, Proposition 2] that

[TABLE]

We will now show that the approximation error of the derivative can be bounded in a similar way. In particular, we show the estimate

[TABLE]

From Equation (C.3) we get for all $k=0,\ldots,2^{m}-1$

[TABLE]

Combining Equation (C.4) and (C.5) yields

[TABLE]

Clearly, the weak derivative of $\Phi_{m}$ is a piecewise constant function, which assumes its maximum on the last piece. Hence,

[TABLE]

Let now $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$ and choose $m=\lceil\log_{2}(\nicefrac{{1}}{{\varepsilon}})\rceil$. Now, $\Phi^{\text{sq}}_{\varepsilon}:=\Phi_{m}$ satisfies the approximation bound in Equation (C.1) and $R_{\varrho}(\Phi^{\text{sq}}_{\varepsilon})(0)=0$. The estimate (C.2) holds because of Equation (C.6). The number of weights can be bounded by

[TABLE]

In the same way, the number of neurons and layers can be bounded. This concludes the proof.

∎

As in [61], we will now use Proposition C.1 and the polarization identity

[TABLE]

to define an approximate multiplication where the approximation error is again (and in contrast to [61]) measured in the $W^{1,\infty}$ norm.

Proposition C.2**.**

For any $M\geq 1$, there exist constants $c,c_{1}>0$ and $c_{2}=c_{2}(M)>0$ such that for any $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$, there is a neural network $\widetilde{\times}$ with two-dimensional input and one-dimensional output that satisfies the following properties:

1. (i)

$\lVert R_{\varrho}(\widetilde{\times})(x,y)-xy\rVert_{W^{1,\infty}(\left(-M,M\right)^{2};dxdy)}\leq\varepsilon$; 2. (ii)

if $x=0$ or $y=0$, then $R_{\varrho}(\widetilde{\times})\left(x,y\right)=0$; 3. (iii)

$\lvert R_{\varrho}(\widetilde{\times}_{\varepsilon})\rvert_{W^{1,\infty}(\left(-M,M\right)^{2})}\leq cM$; 4. (iv)

the depth and the number of weights and neurons in $\widetilde{\times}$ is at most $c_{1}\log_{2}(1/\varepsilon)+c_{2}$.

Proof.

Set $\delta:=\varepsilon/(6M^{2}C)$, where $C$ is the constant from Corollary B.5 for $n=2$ and $m=1$, and let $\Phi^{\text{sq}}_{\delta}$ be the approximate squaring network from Proposition C.1 such that

[TABLE]

As in the proof of [61, Proposition 3], we use the fact that $\lvert x\rvert=\varrho(x)+\varrho(-x)$ to see that a network $\widetilde{\times}$ can be constructed with two-dimensional input and one-dimensional output that satisfies

[TABLE]

As a consequence of Proposition C.1, there exists a constant $c_{0}>0$ such that $\widetilde{\times}$ has at most $c_{0}\log_{2}(\nicefrac{{1}}{{\varepsilon}})+c_{0}\log_{2}(6M^{2})+3\leq c_{0}\log_{2}(\nicefrac{{1}}{{\varepsilon}})+c_{1}$ layers, $3c_{0}\log_{2}(\nicefrac{{1}}{{\varepsilon}})+3c_{0}\log_{2}(6M^{2})+9\leq c^{\prime}\log_{2}(\nicefrac{{1}}{{\varepsilon}})+c_{2}$ neurons and $3c_{0}\log_{2}(\nicefrac{{1}}{{\varepsilon}})+3c_{0}\log_{2}(6M^{2})+17\leq c^{\prime\prime}\log_{2}(\nicefrac{{1}}{{\varepsilon}})+c_{3}$ nonzero weights. Here $c^{\prime},c^{\prime\prime}>0$ and $c_{1}=c_{1}(M),c_{2}=c_{2}(M),c_{3}=c_{3}(M)>0$ are suitable constants and (iv) is hence satisfied.

Since $R_{\varrho}(\Phi^{\text{sq}}_{\delta})(0)=0$, (ii) easily follows.

Using the polarization identity (C.7), we can write

[TABLE]

To keep the following calculations simple, we introduce some notation. In particular, we define

[TABLE]

[TABLE]

and

[TABLE]

Setting $f:\left(0,1\right)\to\mathbb{R}$, $x\mapsto x^{2}$ and using (C.9) we get

[TABLE]

Now, we can estimate

[TABLE]

Note that for the inner functions in the compositions in Equation (C.11), it holds that

[TABLE]

Hence, to finally prove (i), we apply Corollary B.5 to (C.11) and get

[TABLE]

where we used (C.8) in the second step.

To show (iii) we use the chain rule estimate from Corollary B.5 and get

[TABLE]

where we used Equation (C.12) in the third step and $C^{\prime}$ is the absolute constant from Equation (C.2) in Proposition C.1. ∎

Next, we construct a partition of unity (in the same way as in [61, Theorem 1]) that can be defined as a product of piecewise linear functions, such that each factor of the product can be realized by a neural network. We will use this product structure in Lemma C.6 together with a generalized version of the approximate multiplication from Proposition C.2 to approximate localized polynomials with ReLU networks.

Lemma C.3**.**

For any $d,N\in\mathbb{N}$ there exists a collection of functions

[TABLE]

with $\phi_{m}:\mathbb{R}^{d}\to\mathbb{R}$ for all $m\in\{0,\ldots,N\}^{d}$ with the following properties:

1. (i)

$0\leq\phi_{m}(x)\leq 1$* for every $\phi_{m}\in\Psi$ and every $x\in\mathbb{R}^{d}$;* 2. (ii)

$\sum_{\phi_{m}\in\Psi}\phi_{m}(x)=1$* for every $x\in[0,1]^{d}$;* 3. (iii)

$\mathrm{supp\leavevmode\nobreak\ }\phi_{m}\subset B_{\nicefrac{{1}}{{N}},\lVert\cdot\rVert_{\ell^{\infty}}}(\nicefrac{{m}}{{N}})$* for every $\phi_{m}\in\Psi$;* 4. (iv)

there exists a constant $c\geq 1$ such that $\lVert\phi_{m}\rVert_{{W^{k,\infty}(\mathbb{R}^{d})}}\leq(c\cdot N)^{k}$ for $k\in\{0,1\}$; 5. (v)

there exist absolute constants $C,c\geq 1$ such that for each $\phi_{m}\in\Psi$ there is a neural network $\Phi_{m}$ with $d$-dimensional input and $d$-dimensional output, with at most three layers, $Cd$ nonzero weights and neurons, that satisfies

[TABLE]

and $\lVert[R_{\varrho}(\Phi_{m})]_{l}\rVert_{{W^{k,\infty}(\left(0,1\right)^{d})}}\leq(cN)^{k}$ for all $l=1,\ldots,d$ and $k\in\{0,1\}$.

Proof.

As in [61], we define the functions

[TABLE]

and $\phi_{m}:\mathbb{R}^{d}\to\mathbb{R}$ as a product of scaled and shifted versions of $\psi$. Concretely, we set

[TABLE]

for $m=(m_{1},\ldots,m_{d})\in\{0,\ldots,N\}^{d}$. Then, (i),(ii) and (iii) follow easily from the definition.

To show (iv), note that $\lVert\phi_{m}\rVert_{L^{\infty}}\leq 1$ follows already from (i). It now suffices to show the claim for the $W^{1,\infty}$ semi-norm. For this, let $l\in\{1,\ldots,d\}$ and $x\in\mathbb{R}^{d}$, then

[TABLE]

It follows that $\lvert\phi_{m}\rvert_{W^{1,\infty}}\leq 3N$.

To show (v), we start by constructing a network $\Phi_{\psi}$ that realizes the function $\psi$. For this we set

[TABLE]

and $\Phi_{\psi}:=((A_{1},b_{1}),(A_{2},b_{2}))$. Then $\Phi_{\psi}$ is a two-layer network with one-dimensional input and one-dimensional output, with $12$ nonzero weights and $6$ neurons such that

[TABLE]

We denote by $\Phi_{m,l}$ the one-layer network with $d$-dimensional input and one-dimensional output, with $2$ nonzero weights and $d+1$ neurons such that $R_{\varrho}(\Phi_{m,l})=3N(x_{l}-m_{l}/N)$ for all $x\in\mathbb{R}^{d}$ and $m\in\{0,\ldots,N\}^{d}$, $l=1,\ldots,d$. Finally, we define

[TABLE]

which is a three-layer network with $d$-dimensional input and $d$-dimensional output, with at most $d\cdot 2\cdot(12+2)=d\cdot 28$ nonzero weights and at most $9d$ neurons, and

[TABLE]

for $l=1,\ldots,d$ and $x\in\mathbb{R}^{d}$. Clearly, it follows that $\prod_{l=1}^{d}[R_{\varrho}(\Phi_{m})]_{l}(x)=\phi_{m}(x)$ for all $x\in\mathbb{R}^{d}$. The last part of (v) can be shown similarly as (iv). ∎

The following lemma uses the partition of unity from Lemma C.3 and the Bramble-Hilbert Lemma B.12 in a classical way to derive an approximation with localized polynomials in the $L^{p}$ norm and the $W^{1,p}$ norm. Using an interpolation argument, we can generalize this result to the case where the approximation is performed with respect to the $W^{s,\infty}$ norm, where $0\leq s\leq 1$.

Lemma C.4**.**

Let $d,N\in\mathbb{N}$, $n\in\mathbb{N}_{\geq 2}$, $1\leq p\leq\infty$ and $\Psi=\Psi(d,N)=\left\{\phi_{m}:m\in\{0,\ldots,N\}^{d}\right\}$ be the partition of unity from Lemma C.3. Then there is a constant $C=C(d,n,p)>0$ such that for any $f\in{W^{n,p}(\left(0,1\right)^{d})}$, there exist polynomials $p_{f,m}(x)=\sum_{|\alpha|\leq n-1}c_{f,m,\alpha}x^{\alpha}$ for $m\in\{0,\ldots,N\}^{d}$ with the following properties:

Let $0\leq s\leq 1$ and set $f_{N}:=\sum_{m\in\{0,\ldots,N\}^{d}}\phi_{m}p_{f,m}$, then the operator $T_{s}:{W^{n,p}(\left(0,1\right)^{d})}\to{W^{s,p}(\left(0,1\right)^{d})}$ with $T_{s}f=f-f_{N}$ is linear and bounded with

[TABLE]

Furthermore, there is a constant $c=c(d,n)>0$ such that for any $f\in{W^{n,p}(\left(0,1\right)^{d})}$ the coefficients of the polynomials $p_{f,m}$ satisfy

[TABLE]

for all $\alpha\in\mathbb{N}^{d}_{0}$ with $\lvert\alpha\rvert\leq n-1$ and $m\in\{0,\ldots,N\}^{d}$, where $\Omega_{m,N}:=B_{\frac{1}{N},\lVert\cdot\rVert_{\ell^{\infty}}}\left(\frac{m}{N}\right)$ and $\tilde{f}\in{W^{n,p}(\mathbb{R}^{d})}$ is an extension of $f$.

Proof.

The idea of the proof is similar to the first part of the proof of [61, Theorem 1]. We use approximation properties of averaged Taylor polynomials (see Bramble-Hilbert Lemma B.12) to derive local estimates and then combine them using a partition of unity to obtain a global estimate. In order to use this strategy also near the boundary, we make use of an extension operator.

For this, let $E:{W^{n,p}(\left(0,1\right)^{d})}\to{W^{n,p}(\mathbb{R}^{d})}$ be the extension operator from [55, Theorem VI.3.1.5] and set $\tilde{f}:=Ef$. Note that for arbitrary $\Omega\subset\mathbb{R}^{d}$ and $1\leq k\leq n$ it holds

[TABLE]

where $C_{E}=C_{E}(n,p,d)$ is the norm of the extension operator.

Step 1 (Averaged Taylor polynomials): For each $m\in\{0,\ldots,N\}^{d}$ we set

[TABLE]

and denote by $p_{m}=p_{f,m}$ the Taylor polynomial of order $n$ of $\tilde{f}$ averaged over $B_{m,N}$ (cf. Definition B.7). It follows from Proposition B.9 (for $\Omega=\Omega_{m,N}$, $B=B_{m,N}$ and $R=2$) that we can write $p_{m}=\sum_{|\alpha|\leq n-1}c_{m,\alpha}x^{\alpha}$ and that there is a constant $c^{\prime}=c^{\prime}(n,d)>0$ such that

[TABLE]

for $m\in\{0,\ldots,N\}^{d}$, where $c^{\prime\prime}=c^{\prime\prime}(n,d,p)>0$ is a suitable constant. It now suffices to show (i) and (ii).

Step 2 (Local estimates in $\lVert\cdot\rVert_{W^{k,p}},k\in\{0,1\}$): To check that the conditions of the Bramble-Hilbert Lemma B.12 are fulfilled, note that $B_{m,N}\subset\subset\Omega_{m,N}$. Furthermore, $B_{m,N}$ is a ball in $\Omega_{m,N}$ such that $\Omega_{m,N}$ is star-shaped with respect to $B_{m,N}$. We have $\operatorname{diam}_{\lvert\cdot\rvert}(\Omega_{m,N})=(2\sqrt{d})/N$ and $r^{\star}_{\text{max}}(\Omega_{m,N})=1/N$ and, thus,

[TABLE]

Finally, we have for the chunkiness parameter of $\Omega_{m,N}$

[TABLE]

Applying the Bramble-Hilbert Lemma B.12 yields for each $m\in\{0,\ldots,N\}^{d}$ the local estimate

[TABLE]

Here, $C_{1}=C_{1}(n,d)>0$ is the constant from Lemma B.12 which only depends on $n$ and $d$, since the chunkiness parameter of $\Omega_{m,N}$ is a constant depending only on $d$ (see (C.15)) and $C_{2}=C_{2}(n,d)>0$. In the same way, we get

[TABLE]

where $C_{3}=C_{3}(n,d)>0$ is a suitable constant.

The first step towards a global estimate is now to combine Equation (C.16) and (C.17) with the cut-off functions from the partition of unity. We have

[TABLE]

Furthermore, using the product inequality for weak derivatives from Lemma B.6 we get that there is a constant $C^{\prime}=C^{\prime}(d,p)>0$ such that

[TABLE]

where the first part of the second inequality follows from Lemma C.3 (iv) with $k=1$ and an absolute constant $\tilde{c}\geq 1$, together with Equation (C.16) and the second part from Lemma C.3 (iv) with $k=0$, together with Equation (C.17). Here, $C_{4}=C_{4}(n,d,p)>0$. Now it easily follows from (C.18) and (C.19) that

[TABLE]

for some constant $C_{5}=C_{5}(n,d,p)>0$.

Step 3 (Global estimate in $\lVert\cdot\rVert_{W^{k,p}},k\in\{0,1\}$): To derive the global estimate, we start by noting that with property (ii) from Lemma C.3 we have

[TABLE]

Using that $\tilde{f}$ is an extension of $f$ on $\left(0,1\right)^{d}$ we can write for $k\in\{0,1\}$

[TABLE]

where the last step follows from $\left(0,1\right)^{d}\subset\bigcup_{\widetilde{m}\in\{0,\ldots,N\}^{d}}\Omega_{\widetilde{m},N}$. Now we obtain for each $\widetilde{m}\in\{0,\ldots,N\}^{d}$

[TABLE]

where the triangle inequality together with the support property (iii) from Lemma C.3 is used in the first step. The second step follows again from Lemma C.3 (iii) and the third step follows from (C.18) for $k=0$ and from (C.20) for $k=1$. Here $C_{6}=C_{6}(n,d,p)>0$ can be chosen independent of $k$ (e.g. $C_{6}:=\max\{C_{2},C_{5}\}$).

Finally, the boundedness claim in (i) and (ii) follows from using the definition of $f_{N}$ and combining Equation (C.22) with Equation (C.23):

[TABLE]

where the last two steps follow from the definition of $\Omega_{\widetilde{m},N}$. Thus, we have

[TABLE]

for $k\in\{0,1\}$, where Equation (C.14) was used in the first and second step. Here $C_{7}=C_{7}(n,d,p)>0$ and $C_{8}=C_{8}(n,d,p)$ are constants. The linearity of $T_{k}$, $k\in\{0,1\}$ is a consequence of the linearity of the averaged Taylor polynomial (cf. Remark B.8).

Step 4 (Interpolation): For $0<s<1$ we use a Banach space interpolation argument. Set

[TABLE]

together with

[TABLE]

Then, we can apply Theorem A.4 to the operator $T$ with $Tf:=f-f_{N}$ and get for a constant $C=C(n,d,p)>0$ that

[TABLE]

where we used Lemma A.3 (ii) to see that $(A_{0},A_{1})_{s,p}={W^{n,p}(\left(0,1\right)^{d})}$. ∎

The following technical lemma lays the foundation for approximating localized monomials with neural networks. Using the notation and statement (v) from Lemma C.3, a localized monomial $\phi_{m}x^{\alpha}$ can be expressed as the product of the output components of a network $\Phi_{(m,\alpha)}$ with a suitable output dimension $n$, i.e.

[TABLE]

Given a network $\Phi$ with $n$-dimensional output, we construct a network $\Psi_{\Phi}$ that approximates the product of the output components of $\Phi$.

Lemma C.5**.**

Let $d,m,K\in\mathbb{N}$ and $N\geq 1$ be arbitrary. Then there is a constant $C=C(m)>0$ such that the following holds:

For any $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$, and any neural network $\Phi$ with $d$-dimensional input and $n$-dimensional output where $n\leq m$, and with number of layers, neurons and weights all bounded by $K$, such that

[TABLE]

there exists a neural network $\Psi_{\varepsilon,\Phi}$ with $d$-dimensional input and one-dimensional output, and with number of layers, neurons and weights all bounded by $KC\log_{2}(\nicefrac{{1}}{{\varepsilon}})$, such that

[TABLE]

for $k\in\{0,1\}$ and some constant $c=c(d,m,k)$. Moreover,

[TABLE]

for $x\in\left(0,1\right)^{d}$.

Proof.

Let $d,K\in\mathbb{N}$ and $N\geq 1$. We show by induction over $m\in\mathbb{N}$ that the statement holds. To make the induction argument easier we will additionally show that $c=c(d,m,k)=m^{1-k}c_{1}^{k}$, where $c_{1}=c_{1}(d,m)>0$, and that the network $\Psi_{\varepsilon,\Phi}$ can be chosen such that the first $L(\Phi)-1$ layers of $\Psi_{\varepsilon,\Phi}$ and $\Phi$ coincide and $\lvert R_{\varrho}(\Psi_{\varepsilon,\Phi})\rvert_{{W^{1,\infty}(\left(0,1\right)^{d})}}\leq C_{1}N$ for a constant $C_{1}=C_{1}(d,m)>0$.

If $m=1$, then we can choose $\Psi_{\varepsilon,\Phi}=\Phi$ for any $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$ and the claim holds.

Now, assume that the claim holds for some $m\in\mathbb{N}$. We show that it also holds for $m+1$. For this, let $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$ and let $\Phi=((A_{1},b_{1}),(A_{2},b_{2}),\dots,(A_{L},b_{L}))$ be a neural network with $d$-dimensional input and $n$-dimensional output, where $n\leq m+1$, and with number of layers, neurons and weights all bounded by $K$, where each $A_{l}$ is an $N_{l}\times\sum_{k=0}^{l-1}N_{k}$ matrix, and $b_{l}\in\mathbb{R}^{N_{l}}$ for $l=1,\ldots L$.

Case 1: If $n\leq m$, then we use the induction hypothesis and get that there is a constant $C_{0}=C_{0}(m)>0$ and a neural network $\Psi_{\varepsilon,\Phi}$ with $d$-dimensional input and one-dimensional output, and at most $KC_{0}\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ layers, neurons and weights such that

[TABLE]

for $k\in\{0,1\}$ and $c_{1}=c_{1}(d,m)$. Moreover,

[TABLE]

for any $x\in\left(0,1\right)^{d}$. Furthermore, we have $\lvert R_{\varrho}(\Psi_{\varepsilon,\Phi})\rvert_{{W^{1,\infty}(\left(0,1\right)^{d})}}\leq C_{1}N$, for $C_{1}=C_{1}(d,m)$.

Case 2: Now, we assume that $n=m+1$ and show the claim for constants $\tilde{C}_{0},\tilde{c}_{1}$ and $\tilde{C}_{1}$ depending on $m+1$, possibly different from the constants $C_{0},c_{1}$ and $C_{1}$ from Case 1, respectively. The maximum of each pair of constants fulfills then the claim for networks with output dimension $n\leq m+1$.

We denote by $\Phi_{m}$ the neural network with $d$-dimensional input and $m$-dimensional output which results from $\Phi$ by removing the last output neuron and corresponding weights. In detail, we write

[TABLE]

where $A_{L}^{(1,m)}$ is a $m\times\sum_{k=0}^{L-1}N_{k}$ matrix and $a_{L}^{(m+1)}$ is a $1\times\sum_{k=0}^{L-1}N_{k}$ vector, and $b_{L}^{(1,m)}\in\mathbb{R}^{m}$ and $b_{L}^{(m+1)}\in\mathbb{R}^{1}$. Now we set

[TABLE]

Using the induction hypothesis and the constants $C_{0},c_{1}$ and $C_{1}$ from Case 1, we get that there is a neural network $\Psi_{\varepsilon,\Phi_{m}}=((A^{\prime}_{1},b^{\prime}_{1}),(A^{\prime}_{2},b^{\prime}_{2}),\dots,(A^{\prime}_{L^{\prime}},b^{\prime}_{L^{\prime}}))$ with $d$-dimensional input and one-dimensional output, and at most $KC_{0}\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ layers, neurons and weights such that

[TABLE]

for $k\in\{0,1\}$. Moreover,

[TABLE]

for any $x\in\left(0,1\right)^{d}$. Furthermore, we can assume that $\lvert R_{\varrho}(\Psi_{\varepsilon,\Phi_{m}})\rvert_{{W^{1,\infty}(\left(0,1\right)^{d})}}\leq C_{1}N$, and that the first $L(\Phi)-1$ layers of $\Psi_{\varepsilon,\Phi_{m}}$ and $\Phi_{m}$ coincide and, thus, also the first $L(\Phi)-1$ layers of $\Psi_{\varepsilon,\Phi_{m}}$ and $\Phi$, i.e. $A_{l}=A^{\prime}_{l}$ for $l=1,\ldots,L(\Phi)-1$.

Now, we add the formerly removed neuron with corresponding weights back to the last layer of $\Psi_{\varepsilon,\Phi_{m}}$. For the resulting network

[TABLE]

it holds that the first $L-1$ layers of $\widetilde{\Psi}_{\varepsilon,\Phi}$ and $\Phi$ coincide, and $\widetilde{\Psi}_{\varepsilon,\Phi}$ is a neural network with two-dimensional output. Note that

[TABLE]

where we used Equation (C.26) for $k=0$. Additionally, we have

[TABLE]

Now, we denote by $\widetilde{\times}$ the network from Proposition C.2 with $M=m+1$ and accuracy $\varepsilon$ and define

[TABLE]

Consequently, $\Psi_{\varepsilon,\Phi}$ has $d$-dimensional input, one-dimensional output and, combining the induction hypothesis with statement (iv) of Proposition C.2 and Remark 2.8, at most

[TABLE]

layers, number of neurons and weights. Here $c^{\prime}$ and $c^{\prime\prime}=c^{\prime\prime}(m+1)$ are the constants from Proposition C.2 (iv) for the choice $M=m+1$ and $C=C(m+1)>0$ is a suitable constant. Clearly, the first $L-1$ layers of $\Psi_{\varepsilon,\Phi}$ and $\Phi$ coincide and for the approximation properties it holds that

[TABLE]

for $k\in\{0,1\}$. We continue by considering the first term of the Inequality (C.28) for $k=0$ and obtain

[TABLE]

where we used Proposition C.2 (i) for the last step. Next, we consider the same term for $k=1$ and apply the chain rule from Corollary B.5. For this, let $\hat{C}=\hat{C}(d)$ be the constant from Corollary B.5 (for $n=d$ and $m=2$). We get

[TABLE]

where we used the induction hypothesis together with $\lvert[R_{\varrho}(\Phi)]_{m+1}\rvert_{{W^{1,\infty}(\left(0,1\right)^{d})}}\leq N$ in the third step, and $C_{1}^{\prime}=C_{1}^{\prime}(d,m+1)>0$ is a suitable constant.

To estimate the second term of (C.28) for $k=0$ we use the induction hypothesis (for $k=0$) and get

[TABLE]

For $k=1$ we apply the product rule from Lemma B.6 together with $\lVert[R_{\varrho}(\Phi)]_{m+1}\rVert_{{L^{\infty}}}\leq 1$ and get

[TABLE]

where we used the induction hypothesis for $k=1$, and $c_{1}^{\prime}=c_{1}^{\prime}(d,m+1)>0$.

Combining (C.28) with (C.29) and (C.31) we have

[TABLE]

and in the same way a combination of (C.28) with (C.30) and (C.32) yields

[TABLE]

where $c_{1}^{\prime\prime}=c_{1}^{\prime\prime}(d,m+1)>0$. Putting together the two previous estimates yields

[TABLE]

for a suitable constant $c_{1}^{\prime\prime\prime}=c_{1}^{\prime\prime\prime}(d,m+1)>0$.

We now show Equation (C.25) for $m+1$. To this end, assume that $[R_{\varrho}(\Phi)]_{l}(x)=0$ for some $l\in\{1,\ldots,m+1\}$ and $x\in\left(0,1\right)^{d}$. If $l\leq m$, then Equation (C.27) implies that

[TABLE]

If $l=m+1$, then we have

[TABLE]

Hence, by application of Proposition C.2, we have

[TABLE]

Finally, we need to show that there is a constant $C_{1}^{\prime\prime}=C_{1}^{\prime\prime}(d,m+1)>0$ such that

[TABLE]

Similarly as in (C.30) we have for a constant $C_{1}^{\prime\prime}=C_{1}^{\prime\prime}(d,m+1)>0$ that

[TABLE]

where Corollary B.5 was used for the second step and $\hat{c}$ is the constant from Proposition C.2 (iii) which together with an argument as in (C.30) implies the third step.

Taking the maximum of the each pair of constants derived in Case 1 and Case 2 concludes the proof. ∎

In the next lemma, we approximate a sum of localized polynomials with a neural network. One of the difficulties is to control the derivative of the localizing functions from the partition of unity.

Lemma C.6**.**

Let $d,N\in\mathbb{N}$, $n\in\mathbb{N}_{\geq 2}$, $1\leq p\leq\infty$, and $0\leq s\leq 1$, and let $\Psi=\Psi(d,N)=\left\{\phi_{m}:m\in\{0,\ldots,N\}^{d}\right\}$ be the partition of unity from Lemma C.3. Then, there are constants $C_{1}=C_{1}(n,d,p,s)>0$ and $C_{2}=C_{2}(n,d),C_{3}=C_{3}(n,d)>0$ with the following properties:

For any $\varepsilon\in\left(0,\nicefrac{{1}}{{2}}\right)$ there is a neural network architecture ${\mathcal{A}}_{\varepsilon}={\mathcal{A}}_{\varepsilon}(d,n,N,\varepsilon)$ with $d$-dimensional input and one-dimensional output, with at most $C_{2}\cdot\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ layers and $C_{3}(N+1)^{d}\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ weights and neurons, such that the following holds: Let $f\in{W^{n,p}(\left(0,1\right)^{d})}$ and $p_{m}(x):=p_{f,m}(x)=\sum_{\lvert\alpha\rvert\leq n-1}c_{m,\alpha}x^{\alpha}$ for $m\in\{0,\ldots,N\}^{d}$ be the polynomials from Lemma C.4, then there is a neural network $\Phi_{P,\varepsilon}$ that has architecture ${\mathcal{A}}_{\varepsilon}$ such that

[TABLE]

Proof.

Step 1 (Approximating localized monomials $\phi_{m}x^{\alpha}$): Let $|\alpha|\leq n-1$ and $m\in\{0,\dots,N\}^{d}$. It is easy to see that there is a neural network $\Phi_{\alpha}$ with $d$-dimensional input and $\lvert\alpha\rvert$-dimensional output, with one layer and at most $n-1$ weights and $n-1+d$ neurons such that

[TABLE]

and

[TABLE]

Let now $\Phi_{m}$ be the neural network and $C,c\geq 1$ the constants from Lemma C.3 (v) and define the network

[TABLE]

Then $\Phi_{m,\alpha}$ has at most $3\leq K_{0}$ layers, $Cd+n-1\leq K_{0}$ nonzero weights, and $Cd+n-1+d\leq K_{0}$ neurons for a suitable constant $K_{0}=K_{0}(n,d)\in\mathbb{N}$, and $\prod_{l=1}^{\lvert\alpha\rvert+d}[R_{\varrho}(\Phi_{m,\alpha})]_{l}(x)=\phi_{m}x^{\alpha}$ for all $x\in\left(0,1\right)^{d}$. Moreover, as a consequence of Lemma C.3 (v) together with Equation (C.35) we have

[TABLE]

To construct an approximation of the localized monomials $\phi_{m}x^{\alpha}$, let $\Psi_{\varepsilon,(m,\alpha)}$ be the neural network provided by Lemma C.5 (with $\Phi_{m,\alpha}$ instead of $\Phi$, $m=n-1+d\in\mathbb{N}$, $K=K_{0}\in\mathbb{N}$ and $cN$ instead of $N$) for $m\in\{0,\ldots,N\}^{d}$ and $\alpha\in\mathbb{N}_{0}^{d},\lvert\alpha\rvert\leq n-1$. There exists a constant $C_{1}=C_{1}(n,d)\geq 1$ such that $\Psi_{\varepsilon,(m,\alpha)}$ has at most $C_{1}\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ layers, number of neurons and weights. Moreover,

[TABLE]

for a constant $c^{\prime}=c^{\prime}(n,d,k)>0$ and $k\in\{0,1\}$, and

[TABLE]

Step 2 (Constructing an architecture capable of approximating sums of localized polynomials): We set

[TABLE]

and define the matrix $A_{\text{sum}}\in\mathbb{R}^{1\times M}$ by $A_{\text{sum}}:=[c_{m,\alpha}:m\in\{0,\ldots,N\}^{d},\alpha\in\mathbb{N}_{0}^{d},\lvert\alpha\rvert\leq n-1]$ and the neural network $\Phi_{\text{sum}}:=((A_{\text{sum}},0))$. Finally, we set

[TABLE]

Then, there are constants $C_{2}=C_{2}(n,d),C_{3}=C_{3}(n,d)>0$ such that $\Phi_{P,\varepsilon}$ is a neural network with $d$-dimensional input and one-dimensional output, with at most $1+C_{1}\log_{2}(\nicefrac{{1}}{{\varepsilon}})\leq C_{2}\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ layers,

[TABLE]

nonzero weights and neurons, and

[TABLE]

Note that the network $\Phi_{P,\varepsilon}$ only depends on $p_{f,m}$ (and thus on $f$) via the coefficients $c_{m,\alpha}$. Now, it is easy to see that there exists a neural network architecture ${\mathcal{A}}_{\varepsilon}={\mathcal{A}}_{\varepsilon}(d,n,N,\varepsilon)$ with $L({\mathcal{A}}_{\varepsilon})\leq C_{2}\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ layers and number of neurons and weights bounded by $C_{3}(N+1)^{d}\log_{2}(\nicefrac{{1}}{{\varepsilon}})$ such that $\Phi_{P,\varepsilon}$ has architecture ${\mathcal{A}}_{\varepsilon}$ for every of choice of coefficients $c_{m,\alpha}$, and hence for every choice of $f$.

Step 3 (Estimating the approximation error in $\lVert\cdot\rVert_{W^{k,p}},k\in\{0,1\}$): For each $m\in\{0,\ldots,N\}^{d}$ we set

[TABLE]

and we get for $k\in\{0,1\}$

[TABLE]

where the last step is a consequence of $\left(0,1\right)^{d}\subset\bigcup_{\widetilde{m}\in\{0,\ldots,N\}^{d}}\Omega_{\widetilde{m},N}$. For $\widetilde{m}\in\{0,\ldots,N\}^{d}$ we have

[TABLE]

where we used the triangle inequality in the first step and Lemma C.4 in the second step. Here $c_{1}=c_{1}(n,d)>0$ is a constant. Next, note that

[TABLE]

where $\lambda$ denotes the Lebesgue measure and $c_{3}=c_{3}(d,p)>0$ is a constant. Combining Equation (C.39) with the last estimate yields

[TABLE]

where we used Lemma C.3 (iii) together with Equation (C.37) in the second step and Equation (C.36) in the last step. Here, $c_{4}=c_{4}(n,d,p)>0$ is a constant. Now, using that $\lvert\{\alpha:\alpha\in\mathbb{N}_{0}^{d},\lvert\alpha\rvert\leq n-1\}\rvert\leq(n-1)^{d}$ and Hölder’s inequality shows

[TABLE]

Combining Equation (C.40) with Equation (C.41) and plugging the result in Equation (C.38) finally yields

[TABLE]

where the last step is the same as Step 3 of the proof of Lemma C.4 and $c_{5}=c_{5}(n,d,p,k)>0$ is a constant. Hence the case $s=0$ and $s=1$ is proven.

Step 4 (Interpolation): To show the general statement for $0\leq s\leq 1$ we use the interpolation inequality from Corollary A.5 together with Equation (C.42) and directly get

[TABLE]

for a constant $c_{6}=c_{6}(n,p,d,s)>0$. This concludes the proof of the lemma. ∎

Finally, we are ready to proof the upper complexity bounds.

Proof of Theorem 4.1.

The proof can be divided into two steps: First, we approximate the function $f$ by a sum of localized polynomials and then approximate this sum by a network.

For the first step, we set

[TABLE]

where $C=C(n,d,p)>0$ is the constant from Corollary C.4. Without loss of generality we may assume that $CB\geq 1$. The same corollary yields that if $\Psi=\Psi(d,N)=\left\{\phi_{m}:m\in\{0,\ldots,N\}^{d}\right\}$ is the partition of unity from Lemma C.3, then there exist polynomials $p_{m}(x)=\sum_{\lvert\alpha\rvert\leq n-1}c_{m,\alpha}x^{\alpha}$ for $m\in\{0,\ldots,N\}^{d}$ such that

[TABLE]

For the second step, let $C_{1}=C_{1}(n,d,p,s)>0$, $C_{2}=C_{2}(n,d)>0$ and $C_{3}=C_{3}(n,d)>0$ be the constants from Lemma C.6 and $\Phi_{P,\varepsilon}$ be the neural network provided by Lemma C.6 with $\varepsilon^{n/(n-s)}/(8CB^{2}C_{1})$ instead of $\varepsilon$. Then $\Phi_{P,\varepsilon}$ has at most

[TABLE]

layers for a constant $C^{\prime}=C^{\prime}(n,d,p,B,s)>0$ and at most

[TABLE]

nonzero weights and neurons. Here $C^{\prime\prime}=C^{\prime\prime}(n,d,p,B,s)$ is a suitable constant and we used $(2CB)/\varepsilon\geq 1$ in the first step. It holds that the architecture of $\Phi_{P,\varepsilon}$ is independent of the function $f$. Furthermore, we have

[TABLE]

where we used the inequality $(x+y)^{\sigma}\leq x^{\sigma}+y^{\sigma}$ for $x,y\geq 0$ and $0\leq\sigma\leq 1$ in the last step. We now continue the above computation using that $s/(n-s)\leq 1$ and $2CB\geq 1$ and therefore $(2CB)^{s/(n-s)}\leq 2CB$ and get

[TABLE]

Combining the previous computations we get

[TABLE]

Using the triangle inequality and Equations (C.44) and (C.45) we finally obtain

[TABLE]

which concludes the proof. ∎

Appendix D Lower bounds for approximations

Lower complexity bounds for approximations in $L^{\infty}$ norm, which correspond to the case $k=0$ in Theorem 4.3, have already been shown in Theorem 4 a) in [61]. For lower complexity bounds of $W^{1,\infty}$ approximations we modify the proof strategy outlined in [61, Theorem 4 a)].

We start by showing an auxiliary result, that is used in the proof of Proposition D.2.

Lemma D.1**.**

Let $d\in\mathbb{N}$ and $\Phi$ be a neural network with $d$-dimensional input and one-dimensional output. Moreover, let $x\in\left(0,1\right)^{d}$ and $\nu\in\mathbb{R}^{d}$. Then, there exists an open set $T=T(x,\nu)\subset\left(0,1\right)^{d}$ and $\delta=\delta(x,\nu,T)>0$ with $x+\lambda\delta\nu\in\overline{T}$ for $0\leq\lambda\leq 1$ and $R_{\varrho}(\Phi)$ is affine-linear on $T$.

Proof.

We start by defining the set $U:=\{x\in\mathbb{R}^{d}:R_{\varrho}(\Phi)\text{ is affine-linear on a neighborhood of }$x$\}$. Standard results on the number of pieces of ReLU neural networks [44] yield that $U$ has only finitely many polyhedral, connected components, $(V_{i})_{i=1}^{k}$ for some $k\in\mathbb{N}$, with $U=\bigcup_{i=1}^{k}V_{i}$ and $\mathbb{R}^{d}=\bigcup_{i=1}^{k}\overline{V_{i}}$. Note that if follows from the definition of $U$ that $V_{i}$ is open for $i=1,\ldots,k$.

Now, set $x_{n}:=x+(1/n)\nu$. By the pigeonhole principle, there exists $q\in\{1,\ldots,k\}$, such that $\overline{V_{q}}$ contains infinitely many $x_{n}$. It is not hard to see that if a closed polyhedron contains a converging sequence on a line, then it also contains a small section of the line including the limit point of the sequence. Thus, there exists $\delta>0$ such that $\{x+\lambda\delta\nu:0\leq\lambda\leq 1\}\subset\overline{V_{q}}\cap\left(0,1\right)^{d}\subset\overline{V_{q}\cap\left(0,1\right)^{d}}$. Then, setting $T:=V_{q}\cap\left(0,1\right)^{d}$ shows the claim. ∎

As in [61], we make use of a combinatorial quantity measuring the expressiveness of a set of binary valued functions $H$ defined on some set $X$, called VC-dimension (see e.g. [3, Chapter 3.3]). We define

[TABLE]

The idea of the next proposition is to relate the approximation error $\varepsilon$ with the number of weights $M({\mathcal{A}}_{\varepsilon})$ of an architecture ${\mathcal{A}}_{\varepsilon}$ capable of realizing such an approximation. To this end, we construct a set of functions $H$ parameterized by elements of $\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})+1}$.

If $w\in\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})}$ and $\delta>0$ is chosen appropriately, then a directional derivative of the function realized by ${\mathcal{A}}_{\varepsilon}(w)$ is computed for the evaluation of $h((w,\delta),\cdot)\in H$. By exploiting the approximation capacity of derivatives of functions realized by ${\mathcal{A}}_{\varepsilon}(w)$ we can find a lower bound for $\operatorname{VCdim}(H)$ depending on $\varepsilon$ (Claim 1).

On the other hand, [3, Theorem 8.4] yields an upper bound of the VC-dimension of $H$ in terms of the number of computations and the dimension of the parameterization of $H$ which can be expressed as a function of $M({\mathcal{A}}_{\varepsilon})$ (Claim 2). Together this gives the desired relation.

Proposition D.2**.**

Let $d\in\mathbb{N}$, $n\in\mathbb{N}_{\geq 2}$ and $B>0$. Then, there are constants $c=c(n,B)>0$ and $C=C(d)$ with the following property:

Let $N\in\mathbb{N},0<\varepsilon\leq cN^{-(n-1)}$ and ${\mathcal{A}}_{\varepsilon}={\mathcal{A}}_{\varepsilon}(d,n,\varepsilon)$ be a neural network architecture with $d$-dimensional input and one-dimensional output such that for any $f\in\mathcal{F}_{n,d,\infty,B}$ there is a neural network $\Phi_{\varepsilon}^{f}$ that has architecture ${\mathcal{A}}_{\varepsilon}$ and

[TABLE]

then

[TABLE]

Proof.

We prove the proposition by showing that there exists a function $h:\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})+1}\times[0,1]^{d}\to\{0,1\}$ with

[TABLE]

To simplify the notation, we set

[TABLE]

Step 1 (Construction of $h$): Let now $0<\varepsilon<c_{1}N^{-(n-1)}/(3\sqrt{d})$ for some constant $c_{1}=c_{1}(n)>0$ to be chosen later, and ${\mathcal{A}}_{\varepsilon}={\mathcal{A}}_{\varepsilon}(d,n,\varepsilon)$ be a neural network architecture as in the claim of the proposition.

For $x\in[0,1]^{d}$ we define a direction $\nu(x)\in\mathbb{R}^{d}$ that points from $x$ into $\left(0,1\right)^{d}$ if $x\in[0,1]^{d}\setminus(0,1)^{d}$ and equals $e_{1}$ if $x\in(0,1)^{d}$. We set

[TABLE]

and define $\nu(x):=\tilde{\nu}(x)/\lvert\tilde{\nu}(x)\rvert$. Moreover, we set $\tilde{x}:=x+\nu(x)/(4N)$ for $x\in[0,1]^{d}$.

To construct $h$ we start be defining a function $g:\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})+1}\times[0,1]^{d}\to\mathbb{R}$ and then, to get a binary valued function, define $h$ by thresholding $g$. In detail, we set

[TABLE]

for $w\in\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})}$, $\delta\in\mathbb{R}$ and $x\in[0,1]^{d}$. Now, we define $h:\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})+1}\times[0,1]^{d}\to\{0,1\}$ by

[TABLE]

for $w\in\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})}$, $\delta\in\mathbb{R}$ and $x\in[0,1]^{d}$.

Claim 1 ($N^{d}\leq\operatorname{VCdim}(H)$): Let $x_{1},\ldots,x_{N^{d}}\in[0,1]^{d}$ such that $\lvert x_{m}-x_{n}\rvert\geq 1/N$ for all $m,n=1,\ldots,N^{d}$ with $m\neq n$ and such that $\tilde{x}_{m}\in\left(0,1\right)^{d}$ for $m=1,\ldots,N^{d}$. Moreover, let $y_{1},\ldots,y_{N^{d}}\in\{0,1\}$ be arbitrary. We aim to construct construct $w_{y}\in\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})}$ and $\delta_{y}\in\mathbb{R}$ with

[TABLE]

To this end, we first define a function $f_{y}\in\mathcal{F}_{n,d,\infty,B}$ with $f_{y}(x_{m})=y_{m}\cdot a$ for some constant $a>0$, and then make use of a neural network $\Phi^{f_{y}}$ that approximates $f_{y}$.

Step 2 (Construction of $f_{y}$): We start by defining a bump function $\psi\in C^{\infty}(\mathbb{R}^{d})$ by

[TABLE]

such that $\psi(0)=1$ and $\mathrm{supp\leavevmode\nobreak\ }\psi\subset B_{1/2,\lvert\cdot\rvert}(0)$. For the derivative $D\psi$ of $\psi$ it holds that there exists a function $\phi:(-\nicefrac{{1}}{{2}},\nicefrac{{1}}{{2}})\to\mathbb{R}_{>0}$ such that444Precisely, $\phi(r)=e^{-\left(1-4r^{2}\right)^{-1}+1}\cdot\frac{8}{(1-4r^{2})^{2}}$.

[TABLE]

Thus, if $x\in\mathbb{R}^{d}$ with $\lvert x\rvert<1/2$, then we have for the derivative of $\psi$ in direction $-x/\lvert x\rvert$ at $x$ that $(D_{-\nicefrac{{x}}{{\lvert x\rvert}}}\psi)(x)=\phi(\lvert x\rvert)\lvert x\rvert>0$ only depends on the norm of $x$.

Next, we define $f_{y}\in C^{\infty}(\mathbb{R}^{d})$ by

[TABLE]

for $x\in\mathbb{R}^{d}$. We have $\lvert f\rvert_{{W^{k,\infty}(\left(0,1\right)^{d})}}\leq BN^{-n}N^{k}\leq B$ for $1\leq k\leq n$ and, consequently, $f\in\mathcal{F}_{n,d,\infty,B}$. Furthermore, for $x\in\mathbb{R}^{d}$ with $\lvert x\rvert<1/(2N)$ it holds that

[TABLE]

and, in particular, if $\lvert x\rvert=1/(4N)$, then

[TABLE]

Here, we defined the constant $c_{1}=c_{1}(n,B)>0$ which was left unspecified in the beginning of the proof by $c_{1}:=\frac{B\phi(1/4)}{4\lVert\psi\rVert_{{W^{n,\infty}(\left(0,1\right)^{d})}}}$.

Step 3 (Existence of $w_{y}$ and $\delta_{y}$): We can find a vector $w_{y}\in\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})}$ such that for the neural network $\Phi_{\varepsilon}^{f_{y}}:={\mathcal{A}}_{\varepsilon}(w_{y})$ Equation (D.1) holds (with ${f_{y}}$ instead of $f$). In particular, we have

[TABLE]

Next, for $x\in\left(0,1\right)^{d}$ and $\nu\in\mathbb{R}^{d}$ we get from Lemma D.1 that there exists an open set $T_{x,\nu}\subset\mathbb{R}^{d}$ and $\delta=\delta(x,\nu,T_{x,\nu})>0$ with $x+\lambda\delta\nu\in\overline{T_{x,\nu}}$ for $0\leq\lambda\leq 1$ and $R_{\varrho}(\Phi_{\varepsilon}^{f_{y}})$ is affine-linear on $T_{x,\nu}$. We define

[TABLE]

Let $m\in\{1,\ldots,N^{d}\}$ and let $F_{m}:\mathbb{R}^{d}\to\mathbb{R}$ be an affine-linear function such that $R_{\varrho}(\Phi_{\varepsilon}^{f_{y}})(x)=F_{m}(x)$ for all $x\in T_{\tilde{x}_{m},-\nu(x_{m})}$. It then follows from the continuity of $R_{\varrho}(\Phi_{\varepsilon}^{f_{y}})$ that $R_{\varrho}(\Phi_{\varepsilon}^{f_{y}})(x)=F_{m}(x)$ for all $x\in\overline{T}_{\tilde{x}_{m},-\nu(x_{m})}$. This, together with the choice of $\delta_{y}$ implies that

[TABLE]

Recall that $T_{\tilde{x}_{m},-\nu(x_{m})}$ is open and $f_{y}$ and $R_{\varrho}(\Phi_{\varepsilon}^{f_{y}})$ are continuously differentiable on $T_{\tilde{x}_{m},-\nu(x_{m})}$. Thus, the weak and strong derivative agree and Equation (D.3) implies

[TABLE]

for all $x\in T_{\tilde{x}_{m},-\nu(x_{m})}$ and $i=1,\ldots,d$. Using the continuity of $D^{i}f_{y}$ we get that $\lvert D^{i}f_{y}(\tilde{x}_{m})-D^{i}F_{m}\rvert\leq\varepsilon$ for $i=1,\ldots,d$ and hence

[TABLE]

An addition of zero yields

[TABLE]

We get for the case $y_{m}=1$ that

[TABLE]

where we used Equation (D.5) together with Equation (D.4) and $\lvert\nu(x_{m})\rvert=1$ for the first step and Equation (D.2) together with the upper bound for $\varepsilon$ for the second step.

In a similar way, we get for the case $y_{m}=0$ that

[TABLE]

where we used that $D_{-\nu(x_{m})}f_{y}(\tilde{x}_{m})=0$.

Finally, combining Equation (D.6) and Equation (D.7) reads as

[TABLE]

which proves Claim 1.

Claim 2 ($\operatorname{VCdim}(H)\leq C\cdot M({\mathcal{A}}_{\varepsilon})^{2}$): We start by showing that there exists a constant $C^{\prime}=C^{\prime}(d)$ such that $h((w,\delta),x)$ can be computed using $C^{\prime}\cdot M({\mathcal{A}}_{\varepsilon})$ operations of the following types

• •

the arithmetic operations $+,-,\times$, and $/$ on real numbers,

• •

jumps conditioned on $>,\geq,<,\leq,=$, and $\neq$ comparisons of real numbers

for all $w\in\mathbb{R}^{M({\mathcal{A}}_{\varepsilon})+1}$ and $x\in[0,1]^{d}$.

There exists an absolute constant $C_{1}>0$ such that at most $C_{1}\cdot d$ operations of the specified type are needed to compute $\nu(x)$. Hence, the same holds true for $\tilde{x}$ and $\tilde{x}-\delta\nu(x)$.

Note that the number of neurons that are needed for the computation of $R_{\varrho}({\mathcal{A}}_{\varepsilon}(w))$ can be bounded by $M({\mathcal{A}}_{\varepsilon})$. Thus, $R_{\varrho}({\mathcal{A}}_{\varepsilon}(w))$ can be computed using at most $C_{2}\cdot M({\mathcal{A}}_{\varepsilon})$ operations where $C_{2}>0$ is an absolute constant. Hence, there exists a constant $C_{3}=C_{3}(d)$ such that for the number of operations of the specified type $t$ needed for the computation of $h(x)$ where $x\in[0,1]^{d}$ it holds that $t\leq C_{3}M({\mathcal{A}}_{\varepsilon})$.

Finally, [3, Theorem 8.4] implies

[TABLE]

where $C_{4}=C_{4}(d)>0$ is a suitable constant. ∎

The proof of the lower complexity bounds is now a simple consequence of Proposition D.2.

Proof of Theorem 4.3.

The case $k=0$ corresponds to [61, Theorem 4 a)].

For the case $k=1$, let $c=c(n,B)$ and $C=C(d)$ be the constants from Proposition D.2 and set

[TABLE]

Then, $N\leq(c/\varepsilon)^{1/(n-1)}$ and, thus, $0<\varepsilon\leq cN^{-(n-1)}$. Now, Proposition D.2 implies that

[TABLE]

We also have $(c/\varepsilon)^{1/(n-1)}\leq 2N$ and hence $c_{1}\varepsilon^{-d/(n-1)}\leq N^{d}$ for a suitable constant $c_{1}=c_{1}(n,B)>0$. Combining this estimate with Equation D.8 yields

[TABLE]

which finally results in

[TABLE]

for a constant $C^{\prime}=C^{\prime}(d,n,B)>0$. ∎