Towards a regularity theory for ReLU networks -- chain rule and global error estimates

TL;DR

This paper develops a rigorous derivative concept for ReLU neural networks that satisfies the chain rule and provides a method to extend local approximation results to global estimates, enhancing understanding of neural network regularity.

Contribution

It introduces a derivative framework compatible with the chain rule for ReLU networks and offers a technique to convert local approximation results into global estimates.

Findings

A new derivative concept satisfying the chain rule for ReLU networks

Method to extend local approximation results to global estimates

Application to high-dimensional PDEs in deep learning

Abstract

Although for neural networks with locally Lipschitz continuous activation functions the classical derivative exists almost everywhere, the standard chain rule is in general not applicable. We will consider a way of introducing a derivative for neural networks that admits a chain rule, which is both rigorous and easy to work with. In addition we will present a method of converting approximation results on bounded domains to global (pointwise) estimates. This can be used to extend known neural network approximation theory to include the study of regularity properties. Of particular interest is the application to neural networks with ReLU activation function, where it contributes to the understanding of the success of deep learning methods for high-dimensional partial differential equations.

Equations

$$\Phi =((A_k,b_k){)}_{k=1}^L,$$

$$\mathcal{R}\Phi =W_L\circ \mathrm{ReLU}\circ W_{L-1}\circ \dots \circ \mathrm{ReLU}\circ W_1,$$

$$\mathrm{ReLU}(x):=(\max \{0,x_1\},\dots ,\max \{0,x_N\})$$

$$\Phi =((A_k,b_k){)}_{k=1}^L,\Psi =(({\tilde{A}}_k,{\tilde{b}}_k){)}_{k=1}^{\tilde{L}}$$

$$\displaystyle\begin{split}&\Psi\odot\Phi:=\\ &\quad\big{(}((A_{k},b_{k}))_{k=1}^{L-1},(\tilde{A}_{1}A_{L},\tilde{A}_{1}b_{L}+\tilde{b}_{1}),((\tilde{A}_{k},\tilde{b}_{k}))_{k=2}^{\tilde{L}}\big{)}.\end{split}$$

$$\mathcal{R}(\Psi \odot \Phi )=\mathcal{R}\Psi \circ \mathcal{R}\Phi .$$

$$(D(u\circ v))(x)=(Du)(v(x))\cdot (Dv)(x).$$

$$H(x):=\text{diag}({\text{mathbbm}1}_{(0,\infty )}(x_1),\dots ,{\text{mathbbm}1}_{(0,\infty )}(x_N))$$

$$\mathcal{D}\Phi :=A_L\cdot H({\mathcal{R}}_{L-1}\Phi )\cdot A_{L-1}\cdot \dots \cdot H({\mathcal{R}}_1\Phi )\cdot A_1.$$

$$(\mathcal{D}\Phi )(x)=(D(\mathcal{R}\Phi ))(x).$$

$$L_i:=\{x\in {\mathbb{R}}^d:w_i(x)=0\}=\{x\in {\mathbb{R}}^d:v_i(x)\le 0\}.$$

$$(D{w}_i)(x)=0\text{for almost every }x\in L_i.$$

$$D{w}_i={\text{mathbbm}1}_{{\mathbb{R}}^d\backslash L_i}\cdot D{v}_i={\text{mathbbm}1}_{(0,\infty )}(v_i)\cdot D{v}_i$$

$$D(\mathrm{ReLU}\circ v)=H(v)\cdot Dv.$$

$$(\mathcal{D}(\Psi \odot \Phi ))(x)=(\mathcal{D}\Psi )(\mathcal{R}\Phi (x))\cdot (\mathcal{D}\Phi )(x).$$

$$\mathcal{D}(\Psi \odot \Phi )=D(\mathcal{R}(\Psi \odot \Phi ))=D(\mathcal{R}\Psi \circ \mathcal{R}\Phi ).$$

$$\displaystyle\lim_{y\to{\mathcal{R}{\Phi}}(x)}\big{[}({\mathcal{D}}\Psi)(y)-({\mathcal{D}}\Psi)({\mathcal{R}{\Phi}}(x))\big{]}\cdot({\mathcal{D}}\Phi)(x)=0.$$

$$\underset{y\to \mathcal{R}\Phi (x)}{\lim }[H(u(y))-H(u(\mathcal{R}\Phi (x)))]\cdot (D(u\circ \mathcal{R}\Phi ))(x)=0.$$

$$\underset{y\to \mathcal{R}\Phi (x)}{\lim }{\text{mathbbm}1}_{(0,\infty )}(u_i(y))={\text{mathbbm}1}_{(0,\infty )}(u_i(\mathcal{R}\Phi (x)))$$

$$(D(u_i\circ \mathcal{R}\Phi ))(x)=0$$

$$(\bar{D}\varrho )(x_i):=\{\begin{array}{ll}0,& x_i\in S\\ (D\varrho )(x_i),& \text{else}\end{array}.$$

$$\{x\in {\mathbb{R}}^d:w_i(x)=s\},s\in S.$$

$${\Vert \left[\mathcal{D}\Psi \circ \mathcal{R}\Phi -Du\circ \mathcal{R}\Phi \right]\mathcal{D}\Phi \Vert }_{L^{\infty }}$$

$$\Vert \mathcal{D}\Psi -Du{\Vert }_{L^{\infty }}\Vert \mathcal{D}\Phi {\Vert }_{L^{\infty }}.$$

$$\Vert f-{\mathcal{R}\Phi }_{\epsilon ,B}{\Vert }_{L^{\infty }(I_B)}\le \epsilon ,$$

$$\Vert Df-\mathcal{D}{\Phi }_{\epsilon ,B}{\Vert }_{L^{\infty }(I_B)}\le c{\epsilon }^r.$$

$$\Vert (Df)(x){\Vert }_2\le \mathbf{c}(1+\Vert x{\Vert }_2^{\kappa }).$$

$$\begin{array}{rl}& \mathcal{R}{\Phi }_B^{\mathrm{char}}(x)=1,x\in I_B,\\ & \mathcal{R}{\Phi }_B^{\mathrm{char}}(x)=0,x\notin I_{B+1}.\end{array}$$

$$B_{\epsilon }\in \mathcal{O}({\epsilon }^{-1})\text{and}{b}_{\epsilon }\in \mathcal{O}({\epsilon }^{-\kappa -1}).$$

$$|f(x)-\mathcal{R}{\Phi }_{\epsilon }(x)|\le \epsilon (1+\Vert x{\Vert }_2^{\kappa +2})$$

Full text

I Introduction

It has been observed that deep neural networks exhibit the remarkable capability of overcoming the curse of dimensionality in a number of different scenarios. In particular, for certain types of high-dimensional partial differential equations (PDEs) there are promising empirical observations [1, 2, 3, 4, 5, 6, 7] backed by theoretical results for both the approximation error [8, 9, 10, 11] as well as the generalization error [12]. In this context it becomes relevant to not only show how well a given function of interest can be approximated by neural networks but also to extend the study to the derivative of this function. A number of recent publications [13, 14, 15] have investigated the required size of a network which is sufficient to approximate certain interesting (classes of) functions within a given accuracy. This is achieved, first, by considering the approximation of basic functions by very simple networks and, subsequently, by combining those networks in order to approximate more difficult structures. To extend this approach to include the regularity of the approximation, one requires some kind of chain rule for the composition of neural networks. For neural networks with differentiable activation function the standard chain rule is sufficient. It, however, fails when considering neural networks with an activation function, which is not everywhere differentiable. Although locally Lipschitz continuous functions are w.r.t the Lebesgue measure almost everywhere (a.e.) differentiable, the standard chain rule is not applicable, as, in general, it does not hold even in an ’almost everywhere’ sense. We will introduce derivatives of neural networks in a way that admits a chain rule which is both rigorous as well as easy to work with. Chain rules for functions which are not everywhere differentiable have been considered in a more general setting in e.g. [16, 17]. We employ the specific structure of neural networks to get stronger results using simpler arguments. In particular it allows for a stability result, i.e. Lemma III.3, the application of which will be discussed in Section V. We would also like to mention a very recent work [18] about approximation in Sobolev norms, where they deal with the issue by using a general bound for the Sobolev norm of the composition of functions from the Sobolev space $W^{1,\infty}$. Note however that this approach leads to a certain factor depending on the dimensions of the domains of the functions, which can be avoided with our method. For ease of exposition, we formulate our results for neural networks with the ReLU activation function. We, however, consider in Section IV how such a chain rule can be obtained for any activation function which is locally Lipschitz continuous (with at most countably many points at which it is not differentiable). In Section V we briefly sketch how the results from Section III can be utilized to get approximation results for certain classes of functions. Subsequently, in Section VI, we present a general method of deriving global error estimates from such approximation results, which are naturally obtained for bounded domains. Ultimately, we discuss how our results can be used to extend known theory, enabling the further study of the approximation of PDE solutions by neural networks.

II Setting

As in [14], we consider a neural network $\Phi$ to be a finite sequence of matrix-vector pairs, i.e.

[TABLE]

where $A_{k}\in{\mathbb{R}}^{N_{k}\times N_{k-1}}$ and $b_{k}\in{\mathbb{R}}^{N_{k}}$ for some depth $L\in{\mathbb{N}}$ and layer dimensions $N_{0},N_{1},\dots,N_{L}\in{\mathbb{N}}$. The realization of the neural network $\Phi$ is the function ${\mathcal{R}{\Phi}}\colon{\mathbb{R}}^{N_{0}}\to{\mathbb{R}}^{N_{L}}$ given by

[TABLE]

where $W_{k}(x)=A_{k}x+b_{k}$ for every $x\in{\mathbb{R}}^{N_{k}}$ and where

[TABLE]

for every $x\in{\mathbb{R}}^{N}$. We distinguish between a neural network and its realization, since $\Phi$ uniquely induces ${\mathcal{R}{\Phi}}$, while in general there can be multiple non-trivially different neural networks with the same realization. The representation of a neural network as a structured set of weights as in (1) allows the introduction of notions of network sizes. While there are slight differences between various publications, commonly considered quantities are the depth (i.e. number of affine transformations), the connectivity (i.e. number of non-zero entries of the $A_{k}$ and $b_{k}$), and the weight bound (i.e. maximum of the absolute values of the entries of the $A_{k}$ and $b_{k}$). In [15] it has been shown that these three quantities determine the length of a bit string which is sufficient to encode the network with a prescribed quantization error. In the following let

[TABLE]

be neural networks with matching dimensions in the sense that ${{\mathcal{R}{\Phi}}\colon{\mathbb{R}}^{d}\to{\mathbb{R}}^{m}}$ and ${{\mathcal{R}{\Psi}}\colon{\mathbb{R}}^{m}\to{\mathbb{R}}^{n}}$. We then define their composition as

[TABLE]

Direct computation shows

[TABLE]

Note that the realization ${\mathcal{R}{\Phi}}$ of a neural network $\Phi$ is continuous piecewise linear (CPL) as a composition of CPL functions. Consequently, it is Lipschitz continuous and the realization ${\mathcal{R}{\Phi}}$ is almost everywhere differentiable by Rademacher’s theorem. In particular all three functions in (6) are a.e. differentiable. This, however, is not sufficient to get the derivative of ${\mathcal{R}{(}}\Psi\odot\Phi)$ from the derivatives of ${\mathcal{R}{\Psi}}$ and ${\mathcal{R}{\Phi}}$ by use of the classical chain rule. Consider the very simple counterexample of $u(x):=\operatorname{ReLU}(x)$ and $v(x):=0$ and formally apply the chain rule, i.e.

[TABLE]

Even though $({D}u)(y)$ is well-defined for every $y\in{\mathbb{R}}\backslash\{0\}$, the expression $({D}u)(v(x))$ is defined for no $x\in{\mathbb{R}}$. In general this problem occurs when the inner function maps a set of positive measure into a set where the derivative of the outer function does not exist. Now in this case, one can directly see that setting $({D}u)(0)$ to any arbitrary value would cause (7) to provide the correct result since $({D}v)(x)=0$.

III ReLU network derivative

We proceed by defining the derivative of an arbitrary neural network in a way such that it not only coincides a.e. with the derivative of the realization, but also admits a chain rule. To this end let $H\colon{\mathbb{R}}^{N}\to{\mathbb{R}}^{N\times N}$ be the function given by

[TABLE]

for every $x=(x_{1},\dots,x_{N})\in{\mathbb{R}}^{N}$ and let $\mathcal{R}_{K}\Phi:={\mathcal{R}{(}}(A_{k},b_{k}))_{k=1}^{K}$. We then define the neural network derivative of $\Phi$ as the function ${\mathcal{D}}\Phi\colon{\mathbb{R}}^{N_{0}}\to{\mathbb{R}}^{N_{L}\times N_{0}}$ given by

[TABLE]

Note that this definition is motivated by formally applying the chain rule with the convention that the derivative of $\max\{0,\,\cdot\,\}$ is zero at the origin. Now we need to verify that this is justified.

Theorem III.1**.**

It holds for almost every $x\in{\mathbb{R}}^{d}$ that

[TABLE]

Proof.

Let $v\colon{\mathbb{R}}^{d}\to{\mathbb{R}}^{N}$ be a locally Lipschitz continuous function, define $w:=\operatorname{ReLU}\circ\,v$, and

[TABLE]

We now use an observation about differentiability on level sets (see e.g. [19, Thm 3.3(i)]), which states that

[TABLE]

As $w_{i}(x)=v_{i}(x)$ for every $x\in{\mathbb{R}}^{d}\backslash L_{i}$, we get a.e.

[TABLE]

and consequently

[TABLE]

The claim follows by induction over the layers $K=1,\dots,L$ of $\Phi$, using (14) with $v=\mathcal{R}_{K}\Phi$ for the induction step. ∎

Note that even for convex ${\mathcal{R}{\Phi}}$ the values of ${\mathcal{D}}\Phi$ on the nullset do not necessarily lie in the respective subdifferentials of ${\mathcal{R}{\Phi}}$, as can be seen in Figure 1. Although Theorem III.1 holds regardless of which value is chosen for the derivative of $\max\{0,\,\cdot\,\}$ at the origin, no choice will guarantee that all values of ${\mathcal{D}}\Phi$ lie in the respective subdifferentials of ${\mathcal{R}{\Phi}}$. Here we have set the derivative at the origin to zero, following the convention of software implementations for deep learning applications, e.g. TensorFlow and PyTorch. Using (LABEL:eq:concDef) and (9) one can verify by direct computation that ${\mathcal{D}}$ obeys the chain rule.

Corollary III.2**.**

It holds for every $x\in{\mathbb{R}}^{d}$ that

[TABLE]

Note that (15) is well-defined as ${\mathcal{D}}\Psi$ exists everywhere, although it only coincides with ${D}({\mathcal{R}{\Psi}})$ almost everywhere. Theorem III.1 however guarantees that we still have a.e.

[TABLE]

Next we provide a technical result dealing with the stability of our chain rule, which will prove to be useful in Section V.

Lemma III.3**.**

It holds for almost every $x\in{\mathbb{R}}^{d}$ that

[TABLE]

Proof.

We first show for every locally Lipschitz continuous function $u\colon{\mathbb{R}}^{m}\to{\mathbb{R}}^{N}$ and for almost every $x\in{\mathbb{R}}^{d}$ that

[TABLE]

If $u_{i}({\mathcal{R}{\Phi}}(x))\neq 0$ we have

[TABLE]

as $u_{i}$ is continuous and ${\mathbbm{1}}_{(0,\infty)}$ is continuous on ${\mathbb{R}}\backslash\{0\}$. Furthermore, [19, Thm 3.3(i)] implies that

[TABLE]

for almost every $x\in{\mathbb{R}}^{d}$ with $u_{i}({\mathcal{R}{\Phi}}(x))=0$. Since a finite union of nullsets is again a nullset, this proves the claim (18). The lemma follows by induction over the layers $K=1,\dots,\tilde{L}$ of $\Psi$ and applying (18) with $u=\mathcal{R}_{K}\Psi$. ∎

IV General Activation Functions

As mentioned in the introduction, it is possible to replace the ReLU activation function in (2) by some locally Lipschitz continuous, component-wise applied function $\varrho\colon{\mathbb{R}}\to{\mathbb{R}}$ with an at most countably large set $S$ of points where $\varrho$ is not differentiable. Specifically, one can define the neural network derivative (with activation function $\varrho$) as in (9) with $\mathbbm{1}_{(0,\infty)}(x_{i})$ in (8) replaced by

[TABLE]

The chain rule can, again, be checked by direct computation and it is straightforward to adapt Theorem III.1 to this more general setting by considering the level sets

[TABLE]

If additionally $\bar{{D}}\varrho$ is continuous on ${\mathbb{R}}\setminus S$, the proof of Lemma III.3 translates without any modifications.

V Utilization in Approximation Theory

These results can now be employed to bound the $L^{\infty}$-norm of ${\mathcal{D}}(\Psi\circ\Phi)-{D}(u\circ\,v)$, given corresponding estimates for the approximation of $u$ and $v$ by $\Psi$ and $\Phi$, respectively. Here, one has to take some care when bounding the term

[TABLE]

by

[TABLE]

Again it can happen that ${\mathcal{R}{\Phi}}$ maps a set of positive measure into a nullset where the estimate for the approximation of ${D}u$ by ${\mathcal{D}}\Psi$ in the essential supremum norm is not valid. However, using the stability result in Lemma III.3 one can for almost every $x\in{\mathbb{R}}^{d}$ shift to a sufficiently close point $y\approx{\mathcal{R}{\Phi}}(x)$ where the estimate holds. In [13] Yarotsky explicitly constructs networks whose realization is a linear interpolation111The interpolation points are uniformly distributed over the domain of approximation and their number grows exponentially with the size of the networks. of the squaring function (see Fig. 1 for illustration), which directly gives an estimate on the approximation rate for the derivatives. These simple networks can then be combined to get networks approximating multiplication, polynomials and eventually, by means of e.g. local Taylor approximation, functions $f$ whose first $n\geq 1$ (weak) derivatives are bounded. This leads to estimates of the form

[TABLE]

with $I_{B}=[-B,B]^{d}$, including estimates for the scaling of the size of the network $\Phi_{\varepsilon,B}$ w.r.t. $B$ and $\varepsilon$. As these constructions are based on composing simpler functions with known estimates one can now employ Theorem III.1 and Corollary III.2 to show that the derivatives of those networks also approximate the derivative of the function, i.e.

[TABLE]

Such constructive approaches can further be found in [8], in [14] for $\beta$-cartoon-like functions, in [20] for $(\bm{b},\varepsilon)$-holomorphic maps, and in [15] for high-frequent sinusoidal functions.

VI Global Error Estimates

The error estimates above are usually only sensible for bounded domains, as the realization of a neural network is always CPL with a finite number of pieces. We briefly discuss a general way of transforming them into global pointwise error estimates, which can be useful in the context of PDEs (see e.g. [9, 10]). In the following assume that we have a function $f$ with an at most polynomially growing derivative, i.e.

[TABLE]

Denote by $\Phi_{B}^{\operatorname{char}}$ a neural network which represents the $d$-dimensional approximate characteristic function of $I_{B}$, i.e. ${\mathcal{R}{\Phi_{B}^{\operatorname{char}}}}(x)\in[0,1]$ and

[TABLE]

See [15, Proof of Thm. VIII.3] for such a construction. Further let $\Phi_{\varepsilon,b}^{\operatorname{mult}}$ be the neural network approximating the multiplication function on $[-b,b]^{2}$ with error $\varepsilon$ (see e.g. [20, Prop. 3.1]).

Now we define the global approximation networks $\Phi_{\varepsilon}$ as the composition of $\Phi_{\varepsilon/2,b_{\varepsilon}}^{\operatorname{mult}}$ with the parallelization of $\Phi_{B_{\varepsilon}}^{\operatorname{char}}$ and $\Phi_{\varepsilon/2,B_{\varepsilon}+1}$ for suitable

[TABLE]

See Figure 2 for an illustration and e.g. [14, Def. 2.7] for a formal definition of parallelization. Considering the errors on $I_{B}$, $I_{B+1}\backslash I_{B}$ and ${\mathbb{R}}^{d}\backslash I_{B+1}$ leads to global estimates, i.e. for every $x\in{\mathbb{R}}^{d}$

[TABLE]

and, by use of the chain rule III.2, for almost every $x\in{\mathbb{R}}^{d}$

[TABLE]

Due to the logarithmic size scaling of the multiplication network, the size of $\Phi_{\varepsilon}$ can be bounded by the size of $\Phi_{\varepsilon/2,B_{\varepsilon}+1}$ plus an additional term in $\mathcal{O}(d+\kappa\log\varepsilon^{-1})$.

VII Application to PDEs

Analyzing the regularity properties of neural networks was motivated by the recent successful application of deep learning methods to PDEs [2, 3, 4, 5, 6, 7, 11]. Initiated by empirical experiments [1] it has been proven that neural networks are capable of overcoming the curse of dimensionality for solving so-called Kolmogorov PDEs [12]. More precisely, the solution to the empirical risk minimization problem over a class of neural networks approximates the solution of the PDE up to error $\varepsilon$ with high probability and with size of the networks and number of samples scaling only polynomially in the dimension $d$ and $\varepsilon^{-1}$. The above requires a suitable learning problem and a sufficiently good approximation of the solution function by neural networks. For Kolmogorov PDEs, this boils down to calculating global Lipschitz coefficients and error estimates for neural networks approximating the initial condition and coefficient functions (see e.g. [9, 10]). Employing estimates of the form (26) one can bound the derivative on $I_{B}$, i.e.

[TABLE]

Using mollification and the mean value theorem we can establish local Lipschitz estimates, i.e. for all $x,y\in(-B,B)^{d}$ that

[TABLE]

and corresponding linear growth bounds

[TABLE]

Similarly, one can use (31) to obtain estimates of the form

[TABLE]

for all $x,y\in{\mathbb{R}}^{d}$ (which are demanded in [10, Theorem 1.1]). Moreover, note that the capability to produce approximation results which include error estimates for the derivative is of significant independent interest. Various numerical methods (for instance Galerkin methods) rely on bounding the error in some Sobolev norm $\|\cdot\|_{W^{1,p}}$, which requires estimates of the derivative differences. We believe that the possibility to obtain regularity estimates significantly contributes to the mathematical theory of neural networks and allows for further advances in the numerical approximation of high dimensional partial differential equations.

VIII Relation to backpropagation in training

The approach discussed here could further be applied to the training of neural networks by (stochastic) gradient descent. Note, however, that this is a slightly different setting. From the approximation theory perspective we were interested in the derivative of $x\mapsto{\mathcal{R}{\Phi}}(x)$, while in training one requires the derivative of $\Phi\mapsto{\mathcal{R}{\Phi}}(x^{*})$ for some fixed sample $x^{*}$. In particular this function is no longer CPL but rather continuous piecewise polynomial. While this would necessitate some technical modifications, we believe that it should be possible to employ the method used here in order to show that the gradient of $\Phi\mapsto{\mathcal{R}{\Phi}}(x^{*})$ coincides a.e. with what is computed by backpropagation using the convention of setting the derivative of $\max\{0,\cdot\}$ to zero at the origin (as well as similar conventions for e.g. max-pooling).

Acknowledgment

The research of JB and DE was supported by the Austrian Science Fund (FWF) under grants I3403-N32 and P 30148.