Simultaneous Neural Network Approximation for Smooth Functions

TL;DR

This paper provides explicit nonasymptotic approximation rates for deep neural networks approximating smooth functions in Sobolev norms, with detailed bounds based on network width and depth, relevant for PDE solvers.

Contribution

It establishes nonasymptotic approximation error bounds for deep ReLU networks approximating smooth functions in Sobolev spaces, explicitly relating error to network size and depth.

Findings

Deep ReLU networks achieve specific approximation rates in Sobolev norms.

Error bounds are explicitly characterized in terms of network width and depth.

Results are motivated by applications to numerical PDE solvers.

Abstract

We establish in this work approximation results of deep neural networks for smooth functions measured in Sobolev norms, motivated by recent development of numerical solvers for partial differential equations using deep neural networks. {Our approximation results are nonasymptotic in the sense that the error bounds are explicitly characterized in terms of both the width and depth of the networks simultaneously with all involved constants explicitly determined.} Namely, for $f\in C^s([0,1]^d)$, we show that deep ReLU networks of width $\mathcal{O}(N\log{N})$ and of depth $\mathcal{O}(L\log{L})$ can achieve a nonasymptotic approximation rate of $\mathcal{O}(N^{-2(s-1)/d}L^{-2(s-1)/d})$ with respect to the $\mathcal{W}^{1,p}([0,1]^d)$ norm for $p\in[1,\infty)$. If either the ReLU function or its square is applied as activation functions to construct deep neural networks of width $\mathcal{O}(N\log{N})$ and of depth $\mathcal{O}(L\log{L})$ to approximate $f\in C^s([0,1]^d)$, the approximation rate is $\mathcal{O}(N^{-2(s-n)/d}L^{-2(s-n)/d})$ with respect to the $\mathcal{W}^{n,p}([0,1]^d)$ norm for $p\in[1,\infty)$.