Deep Network Approximation Characterized by Number of Neurons

TL;DR

This paper provides a detailed quantitative analysis of how the approximation capabilities of deep ReLU neural networks depend on the number of neurons, depth, and the domain's geometry, including irregular and low-dimensional manifolds.

Contribution

It offers a constructive characterization of the approximation rates of deep neural networks for various classes of functions and domains, extending previous results to irregular and low-dimensional manifolds.

Findings

ReLU FNNs can approximate Hölder continuous functions with nearly optimal rates.

Approximation rates depend on the number of neurons, depth, and domain complexity.

Extension of approximation analysis to irregular domains and low-dimensional manifolds.

Abstract

This paper quantitatively characterizes the approximation power of deep feed-forward neural networks (FNNs) in terms of the number of neurons. It is shown by construction that ReLU FNNs with width $\mathcal{O}\big(\max\{d\lfloor N^{1/d}\rfloor,\, N+1\}\big)$ and depth $\mathcal{O}(L)$ can approximate an arbitrary H\"older continuous function of order $\alpha\in (0,1]$ on $[0,1]^d$ with a nearly tight approximation rate $\mathcal{O}\big(\sqrt{d} N^{-2\alpha/d}L^{-2\alpha/d}\big)$ measured in $L^p$-norm for any $N,L\in \mathbb{N}^+$ and $p\in[1,\infty]$. More generally for an arbitrary continuous function $f$ on $[0,1]^d$ with a modulus of continuity $\omega_f(\cdot)$, the constructive approximation rate is $\mathcal{O}\big(\sqrt{d}\,\omega_f( N^{-2/d}L^{-2/d})\big)$. We also extend our analysis to $f$ on irregular domains or those localized in an $\varepsilon$-neighborhood of a $d_{\mathcal{M}}$-dimensional smooth manifold $\mathcal{M}\subseteq [0,1]^d$ with $d_{\mathcal{M}}\ll d$. Especially, in the case of an essentially low-dimensional domain, we show an approximation rate $\mathcal{O}\big(\omega_f(\tfrac{\varepsilon}{1-\delta}\sqrt{\tfrac{d}{d_\delta}}+\varepsilon)+\sqrt{d}\,\omega_f(\tfrac{\sqrt{d}}{(1-\delta)\sqrt{d_\delta}}N^{-2/d_\delta}L^{-2/d_\delta})\big)$ for ReLU FNNs to approximate $f$ in the $\varepsilon$-neighborhood, where $d_\delta=\mathcal{O}\big(d_{\mathcal{M}}\tfrac{\ln (d/\delta)}{\delta^2}\big)$ for any $\delta\in(0,1)$ as a relative error for a projection to approximate an isometry when projecting $\mathcal{M}$ to a $d_{\delta}$-dimensional domain.

Figures

Tables

Table 1. Table 1 : Main steps of the proof of Theorem 1.2 . Step 1 1 1 : dimension reduction via the nearly isometric projection operator 𝑨 𝑨 \bm{A} provided by Theorem 4.4 to obtain an “equivalent” function f ~ ~ 𝑓 \widetilde{f} of f 𝑓 f in a low-dimensional domain using 𝒙 𝒚 = 𝒮 ​ ℒ ​ ( { 𝒙 ∈ ℳ ε : 𝑨 ​ 𝒙 = 𝒚 } ) subscript 𝒙 𝒚 𝒮 ℒ conditional-set 𝒙 subscript ℳ 𝜀 𝑨 𝒙 𝒚 \bm{x}_{\bm{y}}={\mathcal{SL}}\left(\{\bm{x}\in\mathcal{M}_{\varepsilon}:\bm{A}\bm{x}=\bm{y}\}\right) . Step 2 2 2 : construct a ReLU FNN to implement ϕ ~ ≈ f ~ ~ italic-ϕ ~ 𝑓 \widetilde{\phi}\approx\widetilde{f} by Theorem 4.3 . Step 3 3 3 : define a ReLU FNN to implement ϕ italic-ϕ \phi in the original high-dimensional domain via the projection 𝑨 𝑨 \bm{A} . Step 4 4 4 : verify that the approximation error of ϕ ≈ f italic-ϕ 𝑓 \phi\approx f satisfies our requirement.

$f\left(𝒙\right)$ for $𝒙\in {ℳ}_{\epsilon }\subseteq {[0,1]}^d$	$\stackrel{\text{Step }\text{4}}{\approx }$	$\varphi \left(𝒙\right):=\tilde{\varphi }\left(𝑨𝒙\right)$ for $𝒙\in {ℳ}_{\epsilon }\subseteq {[0,1]}^d$
Step $1$ $\Updownarrow {𝒙}_{𝒚}=𝒮ℒ\left(\{𝒙\in {ℳ}_{\epsilon }:𝑨𝒙=𝒚\}\right)$		Step $3$ $\Updownarrow 𝒚=𝑨𝒙$
$\tilde{f}\left(𝒚\right):=f\left({𝒙}_{𝒚}\right)$ for $𝒚\in 𝑨\left({ℳ}_{\epsilon }\right)\subseteq {ℝ}^{d_{\delta }}$	$\stackrel{\text{Step }\text{2}}{\approx }$	$\tilde{\varphi }\left(𝒚\right)$ for $𝒚\in 𝑨\left({ℳ}_{\epsilon }\right)\subseteq {ℝ}^{d_{\delta }}$

Table 2. Table 2 : Time complexity of one training iteration for an FNN of width N 𝑁 N and depth L 𝐿 L .

Number of cores $p$	Time Complexity
	Evaluating ${\varphi }_{𝜽}\left(𝒙\right)$	Evaluating $\frac{\partial {\varphi }_{𝜽}\left(𝒙\right)}{\partial 𝜽}$
$p\in [1,N]$	$𝒪\left(N^{2}L/p\right)$	$𝒪\left(N^{2}L/p\right)$
$p\in (N,N^2]$	$𝒪\left(L\left(N^2/p+\ln \frac{p}{N}\right)\right)$	$𝒪\left(L\left(N^2/p+\ln \frac{p}{N}\right)\right)$
$p\in (N^2,\infty )$	$𝒪\left(L\ln N\right)$	$𝒪\left(L\ln N\right)$

Equations

$$\{\begin{array}{ll}2\lambda N^{-2\alpha },& \text{when L≥2 and d=1,}\\ 2(2\sqrt{d}{)}^{\alpha }\lambda N^{-2\alpha /d},& \text{when L≥3 and d≥2},\end{array}$$

$$\omega_{f}(r)\coloneq\sup\big{\{}|f(\bm{x})-f(\bm{y})|:\bm{x},\bm{y}\in[0,1]^{d},\ \|\bm{x}-\bm{y}\|_{2}\leq r\big{\}},\quad\textnormal{for any $r\geq 0$}.$$

$$\Vert f-\varphi {\Vert }_{L^p([0,1{]}^d)}\le 19\sqrt{d}{\omega }_f(N^{-2/d}{L}^{-2/d}),$$

$$\mathcal{M}_{\varepsilon}\coloneqq\big{\{}{\bm{x}}\in[0,1]^{d}:\inf\{\|{\bm{x}}-{\bm{y}}\|_{2}:{\bm{y}}\in\mathcal{M}\}\leq\varepsilon\big{\}},\quad\textnormal{for $\varepsilon\in(0,1)$.}$$

$$2\omega_{f}\big{(}\tfrac{2\varepsilon}{1-\delta}\sqrt{\tfrac{d}{d_{\delta}}}+2\varepsilon\big{)}+19\sqrt{d}\,\omega_{f}(\tfrac{2\sqrt{d}}{(1-\delta)\sqrt{d_{\delta}}}N^{-2/d_{\delta}}L^{-2/d_{\delta}})$$

$$(1-\delta )|{\mathbf{x}}_1-{\mathbf{x}}_2|\le |\mathbf{A}{\mathbf{x}}_1-\mathbf{A}{\mathbf{x}}_2|\le (1+\delta )|{\mathbf{x}}_1-{\mathbf{x}}_2|,\text{for any }{\mathbf{x}}_1,{\mathbf{x}}_2\in \mathcal{M},$$

$$\scalebox{0.9325}[0.9325]{$\displaystyle|f({\bm{x}})-\phi({\bm{x}})|\leq 2\omega_{f}\big{(}\tfrac{2\varepsilon}{1-\delta}\sqrt{\tfrac{d}{d_{\delta}}}+2\varepsilon\big{)}+19\sqrt{d}\,\omega_{f}(\tfrac{2\sqrt{d}}{(1-\delta)\sqrt{d_{\delta}}}N^{-2/d_{\delta}}L^{-2/d_{\delta}})$},$$

$$\|{\bm{x}}\|_{p}\coloneqq\big{(}|x_{1}|^{p}+|x_{2}|^{p}+\cdots+|x_{d}|^{p}\big{)}^{1/p}.$$

$$\Omega([0,1]^{d},K,\delta)\coloneqq\bigcup_{i=1}^{d}\Big{\{}{\bm{x}}=[x_{1},x_{2},\cdots,x_{d}]^{T}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\in[0,1]^{d}}:x_{i}\in\cup_{k=1}^{K-1}(\tfrac{k}{K}-\delta,\tfrac{k}{K})\Big{\}}.$$

$$\mathcal{NN}(\text{\#input}=d;\text{widthvec}=[100,100])=\mathcal{NN}(\text{\#input}=d;\text{widthvec}=[100{]}^2).$$

$$\mathbf{x}={\tilde{\mathbf{h}}}_0{\text{vbox}\text{scalebox}1.0[1.0]\text{leavevmode}to45.04pt\text{vbox}to16.57pt\text{pgfpicture}\text{makeatletter}\text{lower}-1.98395ptto0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.4pt\text{pgfsys@invoke}\text{nullfont}to0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.72pt\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@setlinewidth}0.576pt\text{pgfsys@setdash}0.0pt\text{pgfsys@roundcap}\text{pgfsys@roundjoin}\text{pgfsys@moveto}-1.48796pt1.98395pt\text{pgfsys@curveto}-1.36395pt1.23996pt0.0pt0.124pt0.37198pt0.0pt\text{pgfsys@curveto}0.0pt-0.124pt-1.36395pt-1.23996pt-1.48796pt-1.98395pt\text{pgfsys@stroke}\text{pgfsys@endscope}\text{pgfsys@moveto}0.0pt14.22638pt\text{pgfsys@lineto}42.53143pt14.22638pt\text{pgfsys@stroke}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@transformcm}1.00.00.01.042.53143pt14.22638pt\text{pgfsys@invoke}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{pgfsys@discardpath}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{lxSVG@closescope}\text{endpgfpicture}}^{{\mathbf{W}}_0,{\mathbf{b}}_0}{\mathbf{h}}_1{⟶}^{\sigma }{\tilde{\mathbf{h}}}_1\cdots {\text{vbox}\text{scalebox}1.0[1.0]\text{leavevmode}to53.73pt\text{vbox}to0.72pt\text{pgfpicture}\text{makeatletter}\text{lower}13.86638ptto0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.4pt\text{pgfsys@invoke}\text{nullfont}to0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.72pt\text{pgfsys@invoke}\text{pgfsys@moveto}0.0pt14.22638pt\text{pgfsys@lineto}52.34767pt14.22638pt\text{pgfsys@stroke}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@transformcm}1.00.00.01.052.34767pt14.22638pt\text{pgfsys@invoke}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{pgfsys@discardpath}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{lxSVG@closescope}\text{endpgfpicture}}^{{\mathbf{W}}_{L-1},{\mathbf{b}}_{L-1}}{\mathbf{h}}_L{⟶}^{\sigma }{\tilde{\mathbf{h}}}_L{\text{vbox}\text{scalebox}1.0[1.0]\text{leavevmode}to43.91pt\text{vbox}to0.72pt\text{pgfpicture}\text{makeatletter}\text{lower}13.86638ptto0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.4pt\text{pgfsys@invoke}\text{nullfont}to0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.72pt\text{pgfsys@invoke}\text{pgfsys@moveto}0.0pt14.22638pt\text{pgfsys@lineto}42.53143pt14.22638pt\text{pgfsys@stroke}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@transformcm}1.00.00.01.042.53143pt14.22638pt\text{pgfsys@invoke}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{pgfsys@discardpath}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{lxSVG@closescope}\text{endpgfpicture}}^{{\mathbf{W}}_L,{\mathbf{b}}_L}{\mathbf{h}}_{L+1}=\varphi (\mathbf{x}),$$

$${\mathbf{h}}_{i+1}={\mathbf{W}}_i\cdot {\tilde{\mathbf{h}}}_i+{\mathbf{b}}_i=:{\mathcal{L}}_i({\tilde{\mathbf{h}}}_i),\text{for i=0,1,⋯,L,}$$

$${\tilde{\mathbf{h}}}_i=\sigma ({\mathbf{h}}_i),\text{for i=1, …, L.}$$

$$\varphi ={\mathcal{L}}_L\circ \sigma \circ {\mathcal{L}}_{L-1}\circ \sigma \circ \cdots \circ \sigma \circ {\mathcal{L}}_1\circ \sigma \circ {\mathcal{L}}_0,$$

$$|f(\mathbf{x})-\varphi (\mathbf{x})|\le 18\sqrt{d}{\omega }_f(N^{-2/d}{L}^{-2/d}),\text{for any x∈[0,1]dΩ([0,1]d,K,δ)},$$

$$|f(\mathbf{x})-\tilde{\varphi }(\mathbf{x})|\le \epsilon ,\text{for any x∈[0,1]dΩ([0,1]d,K,δ),}$$

$$|f(\mathbf{x})-\varphi (\mathbf{x})|\le \epsilon +d\cdot {\omega }_f(\delta ),\text{for any x∈[0,1]d.}$$

$$\begin{split}Kd\delta\big{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}2}|f({\bm{0}})|+{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}2}\omega_{f}(\sqrt{d})\big{)}^{p}&=\lfloor N^{1/d}\rfloor^{2}\lfloor L^{2/d}\rfloor d\delta\big{(}2|f({\bm{0}})|+2\omega_{f}(\sqrt{d})\big{)}^{p}\\ &\leq\big{(}\omega_{f}(N^{-2/d}L^{-2/d})\big{)}^{p}.\end{split}$$

$$\max\big{\{}4d\lfloor N^{1/d}\rfloor+3d,\,12N+8\big{\}}\leq 12\max\big{\{}d\lfloor N^{1/d}\rfloor,\,N+1\big{\}}$$

$$|f(\mathbf{x})-\varphi (\mathbf{x})|\le 18\sqrt{d}{\omega }_f(N^{-2/d}{L}^{-2/d}),\text{for any x∈[0,1]dΩ([0,1]d,K,δ)},$$

$$\begin{split}\|f-\phi\|_{L^{p}([0,1]^{d})}^{p}&=\int_{\Omega([0,1]^{d},K,\delta)}|f({\bm{x}})-\phi({\bm{x}})|^{p}\textnormal{d}{\bm{x}}+\int_{[0,1]^{d}\backslash\Omega([0,1]^{d},K,\delta)}|f({\bm{x}})-\phi({\bm{x}})|^{p}\textnormal{d}{\bm{x}}\\ &\leq Kd\delta\big{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}2}|f({\bm{0}})|+{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}2}\omega_{f}(\sqrt{d})\big{)}^{p}+\big{(}18\sqrt{d}\,\omega_{f}(N^{-2/d}L^{-2/d})\big{)}^{p}\\ &\leq\big{(}\omega_{f}(N^{-2/d}L^{-2/d})\big{)}^{p}+\big{(}18\sqrt{d}\,\omega_{f}(N^{-2/d}L^{-2/d})\big{)}^{p}\\ &\leq\big{(}19\sqrt{d}\,\omega_{f}(N^{-2/d}L^{-2/d})\big{)}^{p}.\end{split}$$

$$d\cdot {\omega }_f(\delta )\le {\omega }_f(N^{-2/d}{L}^{-2/d}).$$

$$|f(\mathbf{x})-\tilde{\varphi }(\mathbf{x})|\le 18\sqrt{d}{\omega }_f(N^{-2/d}{L}^{-2/d}):=\epsilon ,\text{for x∈[0,1]dΩ([0,1]d,K,δ)},$$

$$3^{d}\Big{(}\max\big{\{}4d\lfloor N^{1/d}\rfloor+3d,\,12N+8\big{\}}+4\Big{)}\leq 3^{d+3}\max\big{\{}d\lfloor N^{1/d}\rfloor,\,N+1\big{\}}$$

$$|f(\mathbf{x})-\varphi (\mathbf{x})|\le \epsilon +d\cdot {\omega }_f(\delta )\le 19\sqrt{d}{\omega }_f(N^{-2/d}{L}^{-2/d}),\text{for any x∈[0,1]d}.$$

$$\underset{\varphi \in \mathcal{NN}(\text{\#input}=d;\text{width}\le N;\text{depth}\le L)}{\inf }\Vert \varphi -f{\Vert }_{L^{\infty }([0,1{]}^d)}\ge C\lambda N^{-(2\alpha /d+\rho )}{L}^{-(2\alpha /d+\rho )}.$$

$$\underset{\varphi \in \mathcal{NN}(\text{\#input}=d;\text{width}\le N;\text{depth}\le L)}{\inf }\Vert \varphi -f{\Vert }_{L^{\infty }([0,1{]}^d\backslash \mathcal{H})}\ge C\lambda N^{-(2\alpha /d+\rho )}{L}^{-(2\alpha /d+\rho )}.$$

$$\Vert f-\varphi {\Vert }_{L^{\infty }([0,1{]}^d\backslash \mathcal{H})}\le C\lambda N^{-(2\alpha /d+\rho )}{L}^{-(2\alpha /d+\rho )}.$$

$$\mathcal{NN}(\textnormal{width}\leq N;\ \textnormal{depth}\leq L)\subseteq\mathcal{NN}\big{(}\textnormal{\#parameter}\leq(LN+d+2)(N+1)\big{)},$$

$$\begin{split}&\quad\textnormal{VCDim}\big{(}\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width}\leq N;\ \textnormal{depth}\leq L)\big{)}\\ &\leq C_{1}(LN+d+2)(N+1)L\ln\big{(}(LN+d+2)(N+1)\big{)}\\ &\eqqcolon b_{u}(N,L).\end{split}$$

Full text

Key words. Deep ReLU Neural Networks, Hölder Continuity, Modulus of Continuity, Approximation Theory, Low-Dimensional Manifold, Parallel Computing.

1 Introduction

The approximation theory of neural networks has been an active research topic in the past few decades. Previously, as a special kind of ridge function approximation, shallow neural networks with one hidden layer and various activation functions (e.g., wavelets pursuits [45, 10], adaptive splines [19, 54], radial basis functions [8, 52, 18, 25, 64], sigmoid functions [29, 44, 37, 7, 41, 38, 14, 13, 15]) were widely discussed and admit good approximation properties, e.g., the universal approximation property [16, 30, 29], lessening the curse of dimensionality [4, 22, 21], and providing attractive approximation rate in nonlinear approximation [19, 45, 10, 54, 18, 25, 64].

The introduction of deep networks with more than one hidden layers has made significant impacts in many fields in computer science and engineering including computer vision [35] and natural language processing [1]. New scientific computing tools based on deep networks have also emerged and facilitated large-scale and high-dimensional problems that were impractical previously [24, 20]. The design of deep ReLU FNNs is the key of such a revolution. These breakthroughs have stimulated broad research topics from different points of views to study the power of deep ReLU FNNs, e.g. in terms of combinatorics [50], topology [6], Vapnik-Chervonenkis (VC) dimension [5, 57, 27], fat-shattering dimension [34, 2], information theory [53], classical approximation theory [16, 30, 4, 66, 61], optimization [32, 51, 33] etc.

Particularly in approximation theory, non-quantitative and asymptotic approximation rates of ReLU FNNs have been proposed for various types of functions. For example, smooth functions [43, 39, 65, 23], piecewise smooth functions [53], band-limited functions [49], continuous functions [66], solutions to partial differential equations [31]. However, to the best of our knowledge, existing theories [43, 49, 65, 39, 47, 62, 53, 66, 23, 17] can only provide implicit formulas in the sense that the approximation error contains an unknown prefactor, or work only for sufficiently large $N$ and $L$ larger than some unknown numbers. For example, [66] estimated an approximation rate $c(d)L^{-2\alpha/d}$ via a narrow and deep ReLU FNN, where $c(d)$ is an unknown number depending on $d$, and $L$ is required to be larger than a sufficiently large unknown number $\mathscr{L}$. For another example, given an approximation error $\varepsilon$, [53] proved the existence of a ReLU FNN with a constant but still unknown number of layers approximating a $C^{\beta}$ function within the target error. These works can be divided into two cases: 1) FNNs with varying width and only one hidden layer [18, 25, 40, 64] (visualized by the region in        in Figure 1); 2) FNNs with a fixed width of ${\mathcal{O}}(d)$ and a varying depth larger than an unknown number $\mathscr{L}$ [43, 66] (represented by the region in        in Figure 1).

As far as we know, the first quantitative and non-asymptotic approximation rate of deep ReLU FNNs was obtained in [61]. Specifically, [61] identified an explicit formulas of the approximation rate

[TABLE]

for ReLU FNNs with an arbitrary width $N\in\mathbb{N}^{+}$ and a fixed depth $L\in\mathbb{N}^{+}$ to approximate a Hölder continuous function $f$ of order $\alpha$ with a Hölder constant $\lambda$ (visualized in the region shown by        in Figure 1). The approximation rate ${\mathcal{O}}(N^{-2\alpha/d})$ is tight in terms of $N$ and increasing $L$ cannot improve the approximation rate in $N$. The success of deep FNNs in a broad range of applications has motivated a well-known conjecture that the depth $L$ has an important role in improving the approximation power of deep FNNs. In particular, a very important question in practice would be, given an arbitrary $L$ and $N$, what is the explicit formula to characterize the approximation error so as to see whether the network is large enough to meet the accuracy requirement. Due to the highly nonlinear structure of deep FNNs, it is still a challenging open problem to characterize $N$ and $L$ simultaneously in the approximation rate.

To answer the question just above, we establish the first framework that is able to quantify the approximation power of deep ReLU FNNs essentially with arbitrary width $N$ and depth $L$, achieving a nearly optimal approximation rate, $19\sqrt{d}\,\omega_{f}(N^{-2/d}L^{-2/d})$, for continuous functions $f\in C([0,1]^{d})$. Our result is based on new analysis techniques merely based on the structure of FNNs and a modified bit extraction technique inspired by [5], instead of designing FNNs to approximate traditional approximation basis like polynomials and splines as in the existing literature [56, 39, 55, 65, 66, 47, 26, 43, 53, 59, 62, 48]. The approximation rate obtained here admits an explicit formula to compute the prefactor when $\omega_{f}(\cdot)$ is known. For example, in the case of Hölder continuous functions of order $\alpha$ with a Hölder constant $\lambda$ (denoted as the class {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))}), $\omega_{f}(r)\leq\lambda r^{\alpha}$ for $r\geq 0$, resulting in the approximation rate $19\sqrt{d}\,\lambda N^{-2\alpha/d}L^{-2\alpha/d}$ as mentioned previously. As a consequence, existing works for the function class $C([0,1]^{d})$ are special cases of our result (see Figure 1 for a comparison).

Our key contributions can be summarized as follows.

Upper bound: We provide a quantitative and non-asymptotic approximation rate $19\sqrt{d}\,\omega_{f}(N^{-2/d}L^{-2/d})$ in terms of width ${\mathcal{O}}(N)$ and depth ${\mathcal{O}}(L)$ for functions in $C([0,1]^{d})$ in Theorem 1.1. 2. 2.

Lower bound: Through the nearly tight VC-dimension bounds of ReLU FNNs [27], we show that the approximation rate $19\sqrt{d}\,\omega_{f}(N^{-2\alpha/d}L^{-2\alpha/d})$ in terms of $N$ and $L$ is nearly optimal for {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))} in Theorem 2.3. 3. 3.

The approximation rate in terms of the width and depth in this paper is more generic and useful than the one characterized by the number of nonzero parameters denoted as $W$ in the literature. First, the characterization in terms of width and depth implies the one in terms of $W$, while it is not true the other way around. Second, our theory can provide practical guidance for choosing network sizes in realistic applications while theories in terms of $W$ cannot tell how large a network should be to guarantee a target accuracy, since there are too many networks of different sizes sharing the same number of parameters but with different accuracies. 4. 4.

Finally, three aspects of neural networks in practice are discussed: 1) neural network approximation in a high-dimensional irregular domain; 2) neural network approximation in the case of a low-dimensional data structure; 3) the optimal ReLU FNN in parallel computation.

Our main result, Theorem 1.1 below, shows that ReLU FNNs with width ${\mathcal{O}}(N)$ and depth ${\mathcal{O}}(L)$ can approximate $f$ with an approximation rate $19\sqrt{d}\,\omega_{f}(N^{-2/d}L^{-2/d})$, where $\omega_{f}(\cdot)$ is the modulus of continuity of $f$ defined via

[TABLE]

Theorem 1.1**.**

Given $f\in C([0,1]^{d})$, for any $L\in\mathbb{N}^{+}$, $N\in\mathbb{N}^{+}$, and $p\in[1,\infty]$, there exists a function $\phi$ implemented by a ReLU FNN with width C_{1}\max\big{\{}d\lfloor N^{1/d}\rfloor,\,N+1\big{\}} and depth $12L+C_{2}$ such that

[TABLE]

*where $C_{1}=12$ and $C_{2}=14$ if $p\in[1,\infty)$; $C_{1}=3^{d+3}$ and $C_{2}=14+2d$ if $p=\infty$. *

When Theorem 1.1 is applied to f\in{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))}, the approximation rate is $19\sqrt{d}\,\lambda N^{-2\alpha/d}L^{-2\alpha/d}$, because $\omega_{f}(r)\leq\lambda r^{\alpha}$ for any $r\geq 0$. An immediate question following the constructive approximation is how much we can improve the approximation rate. In fact, the approximation rate of f\in{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))} is asymptotically tight based on VC-dimension as we shall see later.

In most real applications of neural networks, though the target function $f$ is defined in a high-dimensional domain, e.g., $[0,1]^{d}$, where $d$ could be tens of thousands or even millions, only the approximation error of $f$ in a neighborhood of a $d_{\mathcal{M}}$-dimensional manifold $\mathcal{M}$ with $d_{\mathcal{M}}\ll d$ is concerned. Hence, we extend Theorem 1.1 to the case when the domain of $f$ is localized in an $\varepsilon$-neighborhood of a compact $d_{\mathcal{M}}$-dimensional Riemannian submanifold $\mathcal{M}\subseteq[0,1]^{d}$ having condition number $1/\tau$, volume $V$, and geodesic covering regularity $\mathcal{R}$. The $\varepsilon$-neighborhood is defined as

[TABLE]

Let d_{\delta}={\mathcal{O}}\left(\tfrac{d_{\mathcal{M}}\ln(dV\mathcal{R}\tau^{-1}\delta^{-1})}{\delta^{2}}\right)={\mathcal{O}}\big{(}d_{\mathcal{M}}\tfrac{\ln(d/\delta)}{\delta^{2}}\big{)} be an integer for any $\delta\in(0,1)$ such that $d_{\mathcal{M}}\leq d_{\delta}\leq d$. We show an approximation rate

[TABLE]

for ReLU FNNs to pointwisely approximate $f$ on $\mathcal{M}_{\varepsilon}$. The key ideas of the proof is the application of Theorem $3.1$ in [3], which provides a nearly isometric projection $\bm{A}\in\mathbb{R}^{d_{\delta}\times d}$ that maps points in $\mathcal{M}\subseteq[0,1]^{d}$ to a $d_{\delta}$-dimensional domain with

[TABLE]

and the application of Theorem 1.1 in this paper, which constructs the desired ReLU FNN with a size depending on $d_{\delta}$ instead of $d$ to lessen the curse of dimensionality. When $\delta$ is closer to $1$, $d_{\delta}$ is closer to $d_{\mathcal{M}}$ but the isometric property of the projection is weakened; when $\delta$ is closer to [math], the isometric property becomes better but $d_{\delta}$ could be larger than $d$, in which case we can simply enforce $d_{\delta}=d$ and choose the identity map as the projection. Hence, $\delta\in(0,1)$ is a parameter to make a balance between isometry and dimension reduction.

Theorem 1.2**.**

Let $f$ be a continuous function on $[0,1]^{d}$ and $\mathcal{M}\subseteq[0,1]^{d}$ be a compact $d_{\mathcal{M}}$-dimensional Riemannian submanifold. For any $N\in\mathbb{N}^{+}$, $L\in\mathbb{N}^{+}$, $\varepsilon\in(0,1)$, and $\delta\in(0,1)$, there exists a function $\phi$ implemented by a ReLU FNN with width 3^{d_{\delta}+3}\max\big{\{}d_{\delta}\lfloor N^{1/d_{\delta}}\rfloor,\,N+1\big{\}} and depth $12L+14+2d_{\delta}$ such that

[TABLE]

for any ${\bm{x}}\in\mathcal{M}_{\varepsilon}$, where $\mathcal{M}_{\varepsilon}$ is defined in Equation (1.2)

The approximation rate of deep neural networks for functions defined precisely on low-dimensional smooth manifolds has been studied in [60] for $C^{2}$ functions and in [9, 11] for Lipschitz continuous functions. Considering that it might be more reasonable to assume data located in a small neighborhood of low-dimensional smooth manifold in real applications, we introduce the $\varepsilon$-neighborhood of the manifold $\mathcal{M}$ in Theorem 1.2. In general, existing results are again asymptotic and they cannot be applied to estimate the approximation accuracy of a ReLU FNN with arbitrarily given width $N$ and depth $L$, since there is no explicit formula without unknown constants to specify the exact error bound. For example, [9] provides an approximation rate $c_{1}\left(NL\right)^{-c_{2}/d_{\delta}}$ with unknown constants (e.g., $c_{1}$ and $c_{2}$) and requires $NL$ greater than an unknown large number. The demand of an explicit error estimation motivates Theorem 1.2 in this paper. When data are concentrating around $\mathcal{M}$, $\varepsilon$ is very small and the dominant term of the approximation error in (1.3) is $19\sqrt{d}\,\omega_{f}(\tfrac{2\sqrt{d}}{(1-\delta)\sqrt{d_{\delta}}}N^{-2/d_{\delta}}L^{-2/d_{\delta}})$ implying that the approximation via deep ReLU FNNs can lessen the curse of dimensionality.

The analysis above provides a general guide for selecting the width and depth of ReLU FNNs to approximate continuous functions, especially when the computation is conducted with parallel computing, which is usually the case in real applications [58, 12]. As we shall see later, when the approximation accuracy and the parallel computing efficiency are considered together, very deep FNNs become less attractive than those with ${\mathcal{O}}(1)$ depth.

The approximation theories in this paper assume that the target function $f$ is fully accessible, making it possible to estimate the approximation error and identify an asymptotically optimal ReLU FNN with a given budget of neurons to minimize the approximation error. In real applications, usually only a limited number of possibly noisy observations of $f$ is available, resulting in a regression problem in statistics. In the latter case, the problem is usually formulated in a stochastic setting with randomly generated noisy observations and the regression error contains mainly two components: bias and variance. The bias is the difference of the expectation of an estimated function and its ground truth $f$. The approximation theories in this paper play an important role in characterizing the power of neural networks when they are applied to solve regression problems by providing a lower bound of the regression bias.

The rest of this paper is organized as follows. We first prove Theorem 1.1 and show its optimality in Section 2 when assuming Theorem 2.1 is true. Next, Theorem 2.1 is proved in Section 3. In Section 4, three aspects of neural networks in practice will be discussed: 1) neural network approximation in a high-dimensional irregular domain; 2) neural network approximation in the case of a low-dimensional data structure; 3) the optimal ReLU FNN in parallel computation. Finally, Section 5 concludes this paper with a short discussion.

2 Approximation of continuous functions

In this section, we prove Theorem 1.1 and discuss its optimality when assume Theorem 2.1 is true. Notations throughout the proof will be summarized in Section 2.1.

2.1 Notations

Let us summarize all basic notations used in this paper as follows.

• •

Matrices are denoted by bold uppercase letters. For instance, $\bm{A}\in\mathbb{R}^{m\times n}$ is a real matrix of size $m\times n$, and $\bm{A}^{T}$ denotes the transpose of $\bm{A}$. Vectors are denoted as bold lowercase letters. For example, \bm{v}=\left[\begin{array}[]{c}v_{1}\\ \vdots\\ v_{d}\end{array}\right]=[v_{1},\cdots,v_{d}]^{T}\in\mathbb{R}^{d} is a column vector with $\bm{v}(i)=v_{i}$ being the $i$-th element. Besides, “[” and “]” are used to partition matrices (vectors) into blocks, e.g., $\bm{A}=\left[\begin{smallmatrix}\bm{A}_{11}&\bm{A}_{12}\\ \bm{A}_{21}&\bm{A}_{22}\end{smallmatrix}\right]$.

• •

For any $p\in[1,\infty)$, the $p$-norm of a vector ${\bm{x}}=[x_{1},x_{2},\cdots,x_{d}]^{T}\in\mathbb{R}^{d}$ is defined by

[TABLE]

• •

Let $\mu(\cdot)$ be the Lebesgue measure.

• •

Let $1_{S}$ be the characteristic function on a set $S$, i.e., $1_{S}$ is equal to $1$ on $S$ and [math] outside of $S$.

• •

The set difference of two sets $A$ and $B$ is denoted by $A\backslash B:=\{x:x\in A,\ x\notin B\}$.

• •

For any $\xi\in\mathbb{R}$, let $\lfloor\xi\rfloor:=\max\{i:i\leq\xi,\ i\in\mathbb{Z}\}$ and $\lceil\xi\rceil:=\min\{i:i\geq\xi,\ i\in\mathbb{Z}\}$.

• •

Assume $\bm{n}\in\mathbb{N}^{d}$, then $f(\bm{n})=\mathcal{O}(g(\bm{n}))$ means that there exists positive $C$ independent of $\bm{n}$, $f$, and $g$ such that $f(\bm{n})\leq Cg(\bm{n})$ when all entries of $\bm{n}$ go to $+\infty$.

• •

Let $\sigma:\mathbb{R}\to\mathbb{R}$ denote the rectified linear unit (ReLU), i.e. $\sigma(x)=\max\{0,x\}$. With the abuse of notations, we define $\sigma:\mathbb{R}^{d}\to\mathbb{R}^{d}$ as \sigma({\bm{x}})=\left[\begin{array}[]{c}\max\{0,x_{1}\}\\ \vdots\\ \max\{0,x_{d}\}\end{array}\right] for any ${\bm{x}}=[x_{1},\cdots,x_{d}]^{T}\in\mathbb{R}^{d}$.

• •

Given $K\in N^{+}$ and $\delta\in(0,\tfrac{1}{K})$, define a trifling region $\Omega([0,1]^{d},K,\delta)$ of $[0,1]^{d}$ as

[TABLE]

In particular, $\Omega([0,1]^{d},K,\delta)=\emptyset$ if $K=1$. See Figure 2 for two examples of trifling regions.

• •

Let $C^{\alpha}([0,1]^{d})$ be the set containing all Hölder continuous functions on $[0,1]^{d}$ of order $\alpha\in(0,1]$. In particular, the $\lambda$-ball in $C^{\alpha}([0,1]^{d})$ is denoted by {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))} for any $\lambda>0$.

• •

We will use $\mathcal{NN}$ to denote a function implemented by a ReLU FNN for short and use Python-type notations to specify a class of functions implemented by ReLU FNNs with several conditions, e.g., $\mathcal{NN}(\textnormal{c}_{1};\ \textnormal{c}_{2};\ \cdots;\ \textnormal{c}_{m})$ is a set of functions implemented by ReLU FNNs satisfying $m$ conditions given by $\{\textnormal{c}_{i}\}_{1\leq i\leq m}$, each of which may specify the number of inputs (#input), the number of outputs (#output), the total number of neurons in all hidden layers (#neuron), the number of hidden layers (depth), the total number of parameters (#parameter), and the width in each hidden layer (width vec), the maximum width of all hidden layers (width), etc. For example, if $\phi\in\mathcal{NN}(\textnormal{\#input}=2;\ \textnormal{width\hskip 0.5ptvec}=[100,100];\ \textnormal{\#output}=1)$, then $\phi$ is a functions satisfies

• –

$\phi$ maps from $\mathbb{R}^{2}$ to $\mathbb{R}$.

• –

$\phi$ can be implemented by a ReLU FNN with two hidden layers and the number of nodes in each hidden layer is $100$.

• •

$[n]^{L}$ is short for $[n,n,\cdots,n]\in\mathbb{N}^{L}$. For example,

[TABLE]

• •

For a function $\phi\in\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width\hskip 0.5ptvec}=[N_{1},N_{2},\cdots,N_{L}];\ \textnormal{\#output}=1)$, if we set $N_{0}=d$ and $N_{L+1}=1$, then the architecture of the network implementing $\phi$ can be briefly described as follows:

[TABLE]

where $\bm{W}_{i}\in\mathbb{R}^{N_{i+1}\times N_{i}}$ and $\bm{b}_{i}\in\mathbb{R}^{N_{i+1}}$ are the weight matrix and the bias vector in the $i$-th (affine) linear transform $\mathcal{L}_{i}$ in $\phi$, respectively, i.e.,

[TABLE]

and

[TABLE]

In particular, $\phi$ can be represented in a form of function compositions as follows

[TABLE]

which has been illustrated in Figure 3.

• •

The expression “an FNN with width $N$ and depth $L$” means

• –

The maximum width of this FNN for all hidden layers is no more than $N$.

• –

The number of hidden layers of this FNN is no more than $L$.

• •

For $\theta\in[0,1)$, suppose its binary representation is $\theta=\sum_{\ell=1}^{\infty}\theta_{\ell}2^{-\ell}$ with $\theta_{\ell}\in\{0,1\}$, we introduce a special notation $\textnormal{bin}\hskip 1.2pt0.\theta_{1}\theta_{2}\cdots\theta_{L}$ to denote the $L$-term binary representation of $\theta$, i.e., $\textnormal{bin}\hskip 1.2pt0.\theta_{1}\theta_{2}\cdots\theta_{L}\coloneqq\sum_{\ell=1}^{L}\theta_{\ell}2^{-\ell}$.

2.2 Proof of Theorem 1.1

We essentially construct piecewise constant functions to approximate continuous functions in the proof. However, it is impossible to construct a piecewise constant function via ReLU FNNs due to the continuity of ReLU FNNs. Thus, we introduce the trifling region $\Omega([0,1]^{d},K,\delta)$, defined in Equation (2.1), and use ReLU FNNs to implement piecewise constant functions outside of the trifling region. To prove Theorem 1.1, we first establish a theorem showing how to construct ReLU FNNs to pointwisely approximate continuous functions except for the trifling region.

Theorem 2.1**.**

Given $f\in C([0,1]^{d})$, for any $L\in\mathbb{N}^{+}$ and $N\in\mathbb{N}^{+}$, there exists a function $\phi$ implemented by a ReLU FNN with width \max\big{\{}4d\lfloor N^{1/d}\rfloor+3d,\,12N+8\big{\}} and depth $12L+14$ such that $\|\phi\|_{L^{\infty}(\mathbb{R}^{d})}\leq|f({\bm{0}})|+\omega_{f}(\sqrt{d})$ and

[TABLE]

where $K=\lfloor N^{1/d}\rfloor^{2}\lfloor L^{2/d}\rfloor$ and $\delta$ is an arbitrary number in $(0,\tfrac{1}{3K}]$.

With Theorem 2.1 that will be proved in Section 3, we can easily prove Theorem 1.1 for the case $p\in[1,\infty)$. In the early version of this paper, which focuses on continuous functions as target functions, we only considered the case $p\in[1,\infty)$ since it was challenging to control the approximation error in the trifling region. Later in [42] when we considered smooth functions as target functions, we invented a technique that can handle the error in the trifling region as in the lemma below. Therefore, we are now able to control the approximation error for $p=\infty$. The results in this paper are for continuous functions, to which the results in [42] are not applicable; the results in [42] characterize how the smoothness of target functions helps to enhance the approximation capacity of ReLU FNNs, which is not addressed in this paper. It is interesting to point out that the approximation rate ${\mathcal{O}}(N^{-2/d}L^{-2/d})$ for continuous functions in this paper is even better than the rate ${\mathcal{O}}((\tfrac{N}{\ln N})^{-2/d}(\tfrac{L}{\ln L})^{-2/d})$ for functions in $C^{1}([0,1]^{d})$ in [42].

Lemma 2.2** (Theorem $2.1$ of [42]).**

Given $\varepsilon>0$, $N,L,K\in\mathbb{N}^{+}$, and $\delta\in(0,\tfrac{1}{3K}]$, assume $f\in C([0,1]^{d})$ and ${\widetilde{\phi}}$ can be implemented by a ReLU FNN with width $N$ and depth $L$. If

[TABLE]

then there exists a function $\phi$ implemented by a new ReLU FNN with width $3^{d}(N+4)$ and depth $L+2d$ such that

[TABLE]

Now we are ready to prove Theorem 1.1 by assuming Theorem 2.1 is true, which will be proved later in Section 3.2.

Proof of Theorem 1.1.

Let us first consider the case $p\in[1,\infty)$. We may assume $f$ is not a constant function since it is a trivial case. Then $\omega_{f}(r)>0$ for any $r>0$. Set $K=\lfloor N^{1/d}\rfloor^{2}\lfloor L^{2/d}\rfloor$ and choose a small $\delta\in(0,\tfrac{1}{3K}]$ such that

[TABLE]

By Theorem 2.1, there exists a function $\phi$ implemented by a ReLU FNN with width

[TABLE]

and depth $12L+14$ such that $\|\phi\|_{L^{\infty}(\mathbb{R}^{d})}\leq|f({\bm{0}})|+\omega_{f}(\sqrt{d})$ and

[TABLE]

It follows from $\mu(\Omega([0,1]^{d},K,\delta))\leq Kd\delta$ and $\|f\|_{L^{\infty}([0,1]^{d})}\leq|f({\bm{0}})|+\omega_{f}(\sqrt{d})$ that

[TABLE]

Hence, $\|f-\phi\|_{L^{p}([0,1]^{d})}\leq 19\sqrt{d}\,\omega_{f}(N^{-2/d}L^{-2/d})$.

Next, let us discuss the case $p=\infty$. Set $K=\lfloor N^{1/d}\rfloor^{2}\lfloor L^{2/d}\rfloor$ and choose a small $\delta\in(0,\tfrac{1}{3K}]$ such that

[TABLE]

By Theorem 2.1, there exists a function ${\widetilde{\phi}}$ implemented by a ReLU FNN with width \max\big{\{}4d\lfloor N^{1/d}\rfloor+3d,\,12N+8\big{\}} and depth $12L+14$ such that

[TABLE]

By Lemma 2.2, there exists a function $\phi$ implemented by a ReLU FNN with width

[TABLE]

and depth $12L+14+2d$ such that

[TABLE]

So we finish the proof. ∎

2.3 Optimality of Theorem 1.1

This section will show that the approximation rate in Theorem 1.1 is nearly tight and there is no room to improve for the function class {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))}. Theorem 2.3 below shows that the approximation rate ${\mathcal{O}}(\omega_{f}(N^{-(2/d+\rho)}L^{-(2/d+\rho)}))$ for any $\rho>0$ is unachievable, implying the approximation rate in Theorem 1.1 is nearly tight for the function class {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))}.

Theorem 2.3**.**

Given any $\rho>0$ and $C>0$, there exists f\in{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))} such that, for any $J_{0}>0$, there exist $N,L\in\mathbb{N}$ with $NL\geq J_{0}$ satisfying

[TABLE]

In fact, we can show a stronger result than Theorem 2.3. Under the same conditions as in Theorem 2.3, for any ${\mathcal{H}}\in[0,1]^{d}$ with $\mu({\mathcal{H}})\leq 2^{-(d+K^{d}+1)}K^{-d}$, where $K=\lfloor(NL)^{2/d+\rho/(2\alpha)}\rfloor$, it can be proved that

[TABLE]

We will prove (2.2) by contradiction, then Theorem 2.3 holds as a consequence. Assuming Equation (2.2) is false, we have the following claim.

Claim 2.4**.**

There exist $\rho>0$ and $C>0$ such that given any f\in{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))}, there exists $J_{0}=J_{0}(\rho,C,f)>0$ such that, for any $N,L\in\mathbb{N}$ with $NL\geq J_{0}$, there exist $\phi\in\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width}\leq N;\ \textnormal{depth}\leq L)$ and ${\mathcal{H}}\in[0,1]^{d}$ with $\mu({\mathcal{H}})\leq 2^{-(d+K^{d}+1)}K^{-d}$, where $K=\lfloor(NL)^{2/d+\rho/(2\alpha)}\rfloor$, satisfying

[TABLE]

Now let us disprove this claim to show Theorem 2.3 and Equation (2.2) are true.

Disproof of Claim 2.4.

Without the loss of generality, we assume $\lambda=1$; in the case of $\lambda\neq 1$, the proof is similar. We will disprove Claim 2.4 using the VC dimension. Recall that the VC dimension of a class of functions is defined as the cardinality of the largest set of points that this class of functions can shatter. Denote the VC dimension of a function set $\mathscr{F}$ by $\textnormal{VCDim}(\mathscr{F})$. By [27] and the fact

[TABLE]

there exists $C_{1}>0$ such that

[TABLE]

Then we will use Claim 2.4 to estimate a lower bound of

[TABLE]

and this lower bound is asymptotically larger than $b_{u}(N,L)$, which leads to a contradiction.

More precisely, we will construct \{f_{\chi}:\chi\in\mathscr{B}\}\subseteq{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{1}(C^{\alpha}([0,1]^{d}))}, which can shatter $b_{\ell}(N,L)\coloneqq K^{d}$ points, where $\mathscr{B}$ is a set defined later. Then by Claim 2.4, there exists $\{\phi_{\chi}:\chi\in\mathscr{B}\}$ such that this set can shatter $b_{\ell}(N,L)$ points. Finally, $b_{\ell}(N,L)=K^{d}=\lfloor(NL)^{2/d+\rho/(2\alpha)}\rfloor^{d}$ is asymptotically larger than b_{u}(N,L)=C_{1}(LN+d+2)(N+1)L\ln\big{(}(LN+d+2)(N+1)\big{)}, which leads to a contradiction. More details can be found below.

Step $1\colon$ Construct \{f_{\chi}:\chi\in\mathscr{B}\}\subseteq{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{1}(C^{\alpha}([0,1]^{d}))} that scatters $b_{\ell}(N,L)$ points.

Divide $[0,1]^{d}$ into $K^{d}$ non-overlapping sub-cubes $\{Q_{\bm{\beta}}\}_{\bm{\beta}}$ as follows:

[TABLE]

for any index vector ${\bm{\beta}}=[\beta_{1},\beta_{2},\cdots,\beta_{d}]^{T}\in\{1,2,\cdots,K\}^{d}$.

Let $Q(\bm{x}_{0},\eta)\subseteq[0,1]^{d}$ be a hypercube, whose center and sidelength are ${\bm{x}}_{0}$ and $\eta$, respectively. Then we define a function $\zeta_{Q}$ on $[0,1]^{d}$ corresponding to $Q=Q(\bm{x}_{0},\eta)\subseteq[0,1]^{d}$ such that:

• •

$\zeta_{Q}(\bm{x}_{0})=(\eta/2)^{\alpha}/2$;

• •

$\zeta_{Q}(\bm{x})=0$ for any ${\bm{x}}\notin Q\backslash\partial Q$, where $\partial Q$ is the boundary of $Q$;

• •

$\zeta_{Q}$ is linear on the line that connects ${\bm{x}}_{0}$ and ${\bm{x}}$, for any ${\bm{x}}\in\partial Q$.

Define

[TABLE]

For each $\chi\in\mathscr{B}$, we define

[TABLE]

where $\zeta_{Q_{\bm{\beta}}}(\bm{x})$ is the associated function introduced just above. It is easy to check that \{f_{\chi}:\chi\in\mathscr{B}\}\subseteq{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{1}(C^{\alpha}([0,1]^{d}))} can shatter $b_{\ell}(N,L)=K^{d}$ points.

Step $2\colon$ Construct $\{\phi_{\chi}:\chi\in\mathscr{B}\}$ that scatters $b_{\ell}(N,L)$ points.

By Claim 2.4, there exist $\rho>0$ and $C_{2}>0$ such that, for any $f_{\chi}\in\{f_{\chi}:\chi\in\mathscr{B}\}$ there exists $J_{\chi}>0$ such that for all $N,L\in\mathbb{N}$ with $NL\geq J_{\chi}$, there exist $\phi_{\chi}\in\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width}\leq N;\ \textnormal{depth}\leq L)$ and ${\mathcal{H}}_{\chi}$ with $\mu({\mathcal{H}}_{\chi})\leq 2^{-(d+K^{d}+1)}K^{-d}$ such that

[TABLE]

Set ${\mathcal{H}}=\cup_{\chi\in\mathscr{B}}{\mathcal{H}}_{\chi}$ and $J_{1}=\max_{\chi\in\mathscr{B}}J_{\chi}$. Then it holds that

[TABLE]

It follows that for all $\chi\in\mathscr{B}$ and $N,L\in\mathbb{N}$ with $NL\geq J_{1}$, we have

[TABLE]

For each index vector ${\bm{\beta}}\in\{1,2,\cdots,K\}^{d}$ and any $\bm{x}\in\tfrac{1}{2}Q_{\bm{\beta}}$, where $\tfrac{1}{2}Q_{\bm{\beta}}$ denotes the cube whose sidelength is half of that of $Q_{\bm{\beta}}$ sharing the same center of $Q_{\bm{\beta}}$, since $Q_{\bm{\beta}}$ has a sidelength $\tfrac{1}{K}=\lfloor(NL)^{2/d+\rho/(2\alpha)}\rfloor^{-1}$, we have

[TABLE]

where $\bm{x}_{Q_{\bm{\beta}}}$ is the center of $Q_{\bm{\beta}}$. For fixed $d$, $\alpha$, and $\rho$, there exists $J_{2}>0$ large enough such that, for any $N,L\in\mathbb{N}$ with $NL\geq J_{2}$, we have

[TABLE]

By Equation (2.5), for any $\bm{\beta}\in\{1,2,\cdots,K\}^{d}$, we have

[TABLE]

which means $(\tfrac{1}{2}Q_{\bm{\beta}})\cap([0,1]^{d}\backslash{\mathcal{H}})$ is not empty. Therefore, there exists ${\bm{x}}_{\bm{\beta}}\in(\tfrac{1}{2}Q_{\bm{\beta}})\cap([0,1]^{d}\backslash{\mathcal{H}})$ for each $\bm{\beta}\in\{1,2,\cdots,K\}^{d}$ such that

[TABLE]

for any $N,L\in\mathbb{N}$ with $NL\geq J_{0}=\max\{J_{1},J_{2}\}$, where the first, the second, and the last inequalities come from (2.7), (2.8), and (2.6), respectively. In other words, for any $\chi\in\mathscr{B}$ and ${\bm{\beta}}\in\{1,2,\cdots,K\}^{d}$, $f_{\chi}({\bm{x}}_{\bm{\beta}})$ and $\phi_{\chi}({\bm{x}}_{\bm{\beta}})$ have the same sign. Then $\{\phi_{\chi}:\chi\in\mathscr{B}\}$ shatters \big{\{}\bm{x_{\beta}}:\bm{\beta}\in\{1,2,\cdots,K\}^{d}\big{\}} since $\{f_{\chi}:\chi\in\mathscr{B}\}$ shatters \big{\{}\bm{x_{\beta}}:\bm{\beta}\in\{1,2,\cdots,K\}^{d}\big{\}} as discussed in Step $1$. Hence,

[TABLE]

for any $N,L\in\mathbb{N}$ with $NL\geq J_{0}$,

Step $3\colon$ Contradiction.

By Equation (2.3) and (2.9), for any $N,L\in\mathbb{N}$ with $NL\geq J_{0}$, we have

[TABLE]

implying that

[TABLE]

which is a contradiction for sufficiently large $N,L\in\mathbb{N}$. So we finish the proof. ∎

By Theorem 2.3, for any $\rho>0$, the approximation rate cannot be better than ${\mathcal{O}}(N^{-(2\alpha/d+\rho)}L^{-(2/\alpha+\rho)})$, if we use FNNs in $\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width}\leq N;\ \textnormal{depth}\leq L)$ to approximate functions in {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))}. By a similar argument, we can show that the approximation rate cannot be ${\mathcal{O}}(N^{-2\alpha/d}L^{-(2/\alpha+\rho)})$ nor ${\mathcal{O}}(N^{-(2\alpha/d+\rho)}L^{-2\alpha/d})$. Hence, the approximation rate in Theorem 1.1 is nearly tight.

3 Proof of Theorem 2.1

In this section, we will prove Theorem 2.1. We first present the key ideas in Section 3.1. Based on two propositions in Section 3.1, the detailed proof is presented in Section 3.2. Finally, the proofs of two propositions in Section 3.1 can be found in Section 3.3 and 3.4.

3.1 Key ideas of proving Theorem 2.1

We will show that an almost piecewise constant function $\phi$ implemented by a ReLU FNN is enough to achieve the desired approximation rate in Theorem 1.1. Given an arbitrary $f\in C([0,1]^{d})$, we introduce a piecewise constant function $f_{p}\approx f$ serving as an intermediate approximant in our construction in the sense that

[TABLE]

The approximation in $f\approx f_{p}$ is a simple and standard technique in constructive approximation. For example, given arbitrary $N$ and $L$, uniformly partition $[0,1]^{d}$ into ${\mathcal{O}}(N^{2}L^{2})$ pieces and define $f_{p}$ using this partition. Then the approximation error of $f_{p}\approx f$ scales like ${\mathcal{O}}(N^{-2/d}L^{-2/d})$. We will address the approximation in $f_{p}\approx\phi$ with the same error scaling and a limited budget of the FNN size, e.g., ${\mathcal{O}}(NL)$ neurons, based on the fact that $f_{p}$ can be approximately implemented by a ReLU FNN in $[0,1]^{d}\backslash\Omega([0,1]^{d},K,\delta)$, where $\Omega([0,1]^{d},K,\delta)$ is the trifling region near the discontinuous locations of $f_{p}$ with an arbitrarily small Lebesgue measure (see Figure 4 for an illustration). The introduction of the trifling region is to ease the construction of a deep ReLU FNN to implement the desired $\phi$, which is a piecewise linear and continuous function, to approximate the discontinuous function $f_{p}$ by removing the difficulty near discontinuous points, essentially smoothing $f_{p}$ by restricting the approximation domain in $[0,1]^{d}\backslash\Omega([0,1]^{d},K,\delta)$.

Now let us discuss the detailed steps of construction. First, divide $[0,1]^{d}$ into a union of important regions $\{Q_{\bm{\beta}}\}_{\bm{\beta}}$ and the trifling region $\Omega([0,1]^{d},K,\delta)$, where each $Q_{\bm{\beta}}$ is associated with a representative $\bm{x}_{\bm{\beta}}\in Q_{\bm{\beta}}$ such that $f(\bm{x}_{\bm{\beta}})=f_{p}(\bm{x}_{\bm{\beta}})$ for each index vector ${\bm{\beta}}\in\{0,1,\dots,K-1\}^{d}$, where $K={\mathcal{O}}(N^{2/d}L^{2/d})$ is the partition number per dimension (see Figure 6 for examples for $d=1$ and $d=2$). Next, we design a vector function ${\bm{\Phi}}_{1}({\bm{x}})$ constructed via {\bm{\Phi}}_{1}({\bm{x}})=\big{[}\phi_{1}(x_{1}),\phi_{1}(x_{2}),\cdots,\phi_{1}(x_{d})\big{]}^{T} to project the whole cube $Q_{\bm{\beta}}$ to a $d$-dimensional index ${\bm{\beta}}$ for each ${\bm{\beta}}$, where each one-dimensional function $\phi_{1}$ is a step function implemented by a ReLU FNN. The final step is to solve a point fitting problem. To be precise, we construct a function $\phi_{2}$ implemented by a ReLU FNN to map ${\bm{\beta}}$ approximately to $f_{p}({\bm{x}}_{\bm{\beta}})=f({\bm{x}}_{\bm{\beta}})$. Then $\phi_{2}\circ{\bm{\Phi}}_{1}({\bm{x}})=\phi_{2}({\bm{\beta}})\approx f_{p}({\bm{x}}_{\bm{\beta}})=f({\bm{x}}_{\bm{\beta}})$ for any ${\bm{x}}\in Q_{\bm{\beta}}$ and each ${\bm{\beta}}$, implying $\phi\coloneqq\phi_{2}\circ{\bm{\Phi}}_{1}\approx f_{p}\approx f$ on $[0,1]^{d}\backslash\Omega([0,1]^{d},K,\delta)$. We would like to point out that we only need to care about the values of $\phi_{2}$ at a set of points $\{0,1,\cdots,K-1\}^{d}$ in the construction of $\phi_{2}$ according to our design $\phi=\phi_{2}\circ{\bm{\Phi}}_{1}$ as illustrated in Figure 5. Therefore, it is unnecessary to care about the values of $\phi_{2}$ sampled outside the set $\{0,1,\cdots,K-1\}^{d}$, which is a key point to ease the design of a ReLU FNN to implement $\phi_{2}$ as we shall see later.

Finally, we discuss how to implement ${\bm{\Phi}}_{1}$ and $\phi_{2}$ by deep ReLU FNNs with width ${\mathcal{O}}(N)$ and depth ${\mathcal{O}}(L)$ using two propositions as we shall prove in Section 3.3 and 3.4 later. We first construct a ReLU FNN with desired width and depth by Proposition 3.1 to implement a one-dimensional step function $\phi_{1}$. Then ${\bm{\Phi}}_{1}$ can be attained via defining

[TABLE]

Proposition 3.1**.**

For any $N,L,d\in\mathbb{N}^{+}$ and $\delta\in(0,\tfrac{1}{3K}]$ with $K=\lfloor N^{1/d}\rfloor^{2}\lfloor L^{2/d}\rfloor$, there exists a one-dimensional function $\phi$ implemented by a ReLU FNN with width $4\lfloor N^{1/d}\rfloor+3$ and depth $4L+5$ such that

[TABLE]

The construction of $\phi_{2}$ is a direct result of Proposition 3.2 below, the proof of which relies on the bit extraction technique in [5].

Proposition 3.2**.**

Given any $\varepsilon>0$ and arbitrary $N,L,J\in\mathbb{N}^{+}$ with $J\leq N^{2}L^{2}$, assume $\{{y}_{j}\geq 0:j=0,1,\cdots,J-1\}$ is a sample set with $|y_{j}-y_{j-1}|\leq\varepsilon$ for $j=1,2,\cdots,J-1$. Then there exists $\phi\in\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 4L+9;\ \textnormal{\#output}=1)$ such that

1. (i)

$|\phi(j)-{y}_{j}|\leq\varepsilon$* for $j=0,1,\cdots,J-1$;* 2. (ii)

$0\leq\phi(x)\leq\max\{y_{j}:j=0,1,\cdots,J-1\}$* for any $x\in\mathbb{R}$.*

With the above propositions ready, let us prove Theorem 2.1 in Section 3.2. We further assume that $\omega_{f}(r)>0$ for any $r>0$, excluding a simple case when $f$ is a constant function.

3.2 Proof of Theorem 2.1

We essentially construct an almost piecewise constant function implemented by a ReLU FNN with ${\mathcal{O}}(NL)$ neurons to approximate $f$. We may $f$ is not a constant since it is a trivial case. Then $\omega_{f}(r)>0$ for any $r>0$. It is clear that $|f(\bm{x})-f({\bm{0}})|\leq\omega_{f}(\sqrt{d})$ for any ${\bm{x}}\in[0,1]^{d}$. Define ${\widetilde{f}}=f-f({\bm{0}})+\omega_{f}(\sqrt{d})$, then $0\leq{\widetilde{f}}(\bm{x})\leq 2\omega_{f}(\sqrt{d})$ for any ${\bm{x}}\in[0,1]^{d}$. Let $M=N^{2}L$, $K=\lfloor N^{1/d}\rfloor^{2}\lfloor L^{2/d}\rfloor$, and $\delta$ be an arbitrary number in $(0,\tfrac{1}{3K}]$.

The proof can be divided into four steps as follows:

Divide $[0,1]^{d}$ into a union of sub-cubes $\{Q_{\bm{\beta}}\}_{\bm{\beta}\in\{0,1,\cdots,K-1\}^{d}}$ and the trifling region $\Omega([0,1]^{d},K,\delta)$, and denote ${\bm{x}}_{\bm{\beta}}$ as the vertex of $Q_{\bm{\beta}}$ with minimum $\|\cdot\|_{1}$ norm; 2. 2.

Construct a sub-network to implement a vector function ${\bm{\Phi}}_{1}$ projecting the whole cube $Q_{\bm{\beta}}$ to the $d$-dimensional index ${\bm{\beta}}$ for each ${\bm{\beta}}$, i.e., ${\bm{\Phi}}_{1}({\bm{x}})={\bm{\beta}}$ for all ${\bm{x}}\in Q_{\bm{\beta}}$; 3. 3.

Construct a sub-network to implement a function $\phi_{2}$ mapping the index ${\bm{\beta}}$ approximately to ${\widetilde{f}}({\bm{x}}_{\bm{\beta}})$. This core step can be further divided into three sub-steps:

1. 3.1.

Construct a sub-network to implement $\psi_{1}$ bijectively mapping the index set $\{0,1,\cdots,K-1\}^{d}$ to an auxiliary set \mathcal{A}_{1}\subseteq\big{\{}\tfrac{j}{2K^{d}}:j=0,1,\cdots,2K^{d}\big{\}} defined later (see Figure 7 for an illustration); 2. 3.2.

Determine a continuous piecewise linear function $g$ with a set of breakpoints $\mathcal{A}_{1}\cup\mathcal{A}_{2}\cup\{1\}$ satisfying: 1) assign the values of $g$ at breakpoints in $\mathcal{A}_{1}$ based on $\{{\widetilde{f}}({\bm{x}}_{\bm{\beta}})\}_{\bm{\beta}}$, i.e., $g\circ\psi_{1}({\bm{\beta}})={\widetilde{f}}({\bm{x}}_{\bm{\beta}})$; 2) assign the values of $g$ at breakpoints in $\mathcal{A}_{2}\cup\{1\}$ to reduce the variation of $g$ for applying Proposition 3.2; 3. 3.3.

Apply Proposition 3.2 to construct a sub-network to implement a function $\psi_{2}$ approximating $g$ well on $\mathcal{A}_{1}\cup\mathcal{A}_{2}\cup\{1\}$. Then the desired function $\phi_{2}$ is given by $\phi_{2}=\psi_{2}\circ\psi_{1}$ satisfying $\phi_{2}({\bm{\beta}})=\psi_{2}\circ\psi_{1}({\bm{\beta}})\approx g\circ\psi_{1}({\bm{\beta}})={\widetilde{f}}({\bm{x}}_{\bm{\beta}})$; 4. 4.

Construct the final target network to implement the desired function $\phi$ such that $\phi({\bm{x}})=\phi_{2}\circ{\bm{\Phi}}_{1}({\bm{x}})+f({\bm{0}})-\omega_{f}(\sqrt{d})\approx{\widetilde{f}}({\bm{x}}_{\bm{\beta}})+f({\bm{0}})-\omega_{f}(\sqrt{d})=f({\bm{x}}_{\bm{\beta}})$ for ${\bm{x}}\in Q_{\bm{\beta}}$.

The details of these steps can be found below.

Step $1\colon$ Divide $[0,1]^{d}$ into $\{Q_{\bm{\beta}}\}_{\bm{\beta}\in\{0,1,\cdots,K-1\}^{d}}$ and $\Omega([0,1]^{d},K,\delta)$.

Define ${\bm{x}}_{\bm{\beta}}\coloneqq{\bm{\beta}}/K$ and

[TABLE]

for each $d$-dimensional index ${\bm{\beta}}=[\beta_{1},\cdots,\beta_{d}]^{T}\in\{0,1,\cdots,K-1\}^{d}$. Recall that $\Omega([0,1]^{d},K,\delta)$ is the trifling region defined in Equation (2.1). Apparently, ${\bm{x}}_{\bm{\beta}}$ is the vertex of $Q_{\bm{\beta}}$ with minimum $\|\cdot\|_{1}$ norm and

[TABLE]

see Figure 6 for illustrations.

Step $2\colon$ Construct ${\bm{\Phi}}_{1}$ mapping ${\bm{x}}\in Q_{\bm{\beta}}$ to ${\bm{\beta}}$.

By Proposition 3.1, there exists $\phi_{1}\in\mathcal{NN}(\textnormal{width}\leq 4\lfloor N^{1/d}\rfloor+3;\ \textnormal{depth}\leq 4L+5)$ such that

[TABLE]

It follows that $\phi_{1}(x_{i})=\beta_{i}$ if ${\bm{x}}=[x_{1},x_{2},\cdots,x_{d}]^{T}\in Q_{\bm{\beta}}$ for each ${\bm{\beta}}=[\beta_{1},\beta_{2},\cdots,\beta_{d}]^{T}$.

By defining

[TABLE]

we have ${\bm{\Phi}}_{1}({\bm{x}})={\bm{\beta}}$ if ${\bm{x}}\in Q_{\bm{\beta}}$ for ${\bm{\beta}}\in\{0,1,\cdots,K-1\}^{d}$.

Step $3\colon$ Construct $\phi_{2}$ mapping ${\bm{\beta}}$ approximately to ${\widetilde{f}}({\bm{x}}_{\bm{\beta}})$.

The construction of the sub-network implementing $\phi_{2}$ is essentially based on Proposition 3.2. To meet the requirements of applying Proposition 3.2, we first define two auxiliary set $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$ as

[TABLE]

and

[TABLE]

Clearly, $\mathcal{A}_{1}\cup\mathcal{A}_{2}\cup\{1\}=\{\tfrac{j}{2K^{d}}:j=0,1,\cdots,2K^{d}\}$ and $\mathcal{A}_{1}\cap\mathcal{A}_{2}=\emptyset$. See Figure 6 for an illustration of $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$. Next, we further divide this step into three sub-steps.

Step $3.1\colon$ Construct $\psi_{1}$ bijectively mapping $\{0,1,\cdots,K-1\}^{d}$ to $\mathcal{A}_{1}$.

Inspired by the binary representation, we define

[TABLE]

Then $\psi_{1}$ is a linear function bijectively mapping the index set $\{0,1,\cdots,K-1\}^{d}$ to

[TABLE]

Step $3.2\colon$ Construct $g$ to satisfy $g\circ\psi_{1}({\bm{\beta}})={\widetilde{f}}({\bm{x}}_{\bm{\beta}})$ and to meet the requirements of applying Proposition 3.2.

Let $g:[0,1]\to\mathbb{R}$ be a continuous piecewise linear function with a set of breakpoints $\left\{\tfrac{j}{2K^{d}}:j=0,1,\cdots,2K^{d}\right\}=\mathcal{A}_{1}\cup\mathcal{A}_{2}\cup\{1\}$ and the values of $g$ at these breakpoints satisfy the following properties:

• •

The values of $g$ at the breakpoints in $\mathcal{A}_{1}$ are set as

[TABLE]

• •

At the breakpoint $1$, let $g(1)={\widetilde{f}}(\bm{1})$, where $\bm{1}=[1,1,\cdots,1]^{T}\in\mathbb{R}^{d}$;

• •

The values of $g$ at the breakpoints in $\mathcal{A}_{2}$ are assigned to reduce the variation of $g$, which is a requirement of applying Proposition 3.2. Note that

[TABLE]

implying the values of $g$ at $\tfrac{i}{K^{d-1}}-\tfrac{K+1}{2K^{d}}$ and $\tfrac{i}{K^{d-1}}$ have been assigned for $i=1,2,\cdots,K^{d-1}$. Thus, the values of $g$ at the breakpoints in $\mathcal{A}_{2}$ can be successfully assigned by letting $g$ linear on each interval $[\tfrac{i}{K^{d-1}}-\tfrac{K+1}{2K^{d}},\,\tfrac{i}{K^{d-1}}]$ for $i=1,2,\cdots,K^{d-1}$, since $\mathcal{A}_{2}\subseteq\cup_{i=1}^{K^{d-1}}[\tfrac{i}{K^{d-1}}-\tfrac{K+1}{2K^{d}},\,\tfrac{i}{K^{d-1}}]$.

Apparently, such a function $g$ exists (see Figure 7 for an example) and satisfies

[TABLE]

and

[TABLE]

Step $3.3\colon$ Construct $\psi_{2}$ approximating $g$ well on $\mathcal{A}_{1}\cup\mathcal{A}_{2}\cup\{1\}$.

Since 2K^{d}=2\big{(}\lfloor N^{1/d}\rfloor^{2}\lfloor L^{2/d}\rfloor\big{)}^{d}\leq 2\big{(}N^{2}L^{2}\big{)}\leq N^{2}\widetilde{L}^{2}, where $\widetilde{L}=2L$, by Proposition 3.2 (set $y_{j}=g(\tfrac{j}{2K^{2}})$ and $\varepsilon=\omega_{f}(\tfrac{\sqrt{d}}{K})>0$ therein), there exists ${\widetilde{\psi}}_{2}\in\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 4\widetilde{L}+9)=\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 8L+9)$ such that

[TABLE]

and

[TABLE]

By defining $\psi_{2}(x)\coloneqq{\widetilde{\psi}}_{2}(2K^{d}x)$ for any $x\in\mathbb{R}$, we have $\psi_{2}\in\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 8L+9)$,

[TABLE]

and

[TABLE]

Let us end Step $3$ by defining the desired function $\phi_{2}$ as $\phi_{2}\coloneqq\psi_{2}\circ\psi_{1}$. Note that $\psi_{1}:\mathbb{R}^{d}\to\mathbb{R}$ is a linear function and $\psi_{2}\in\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 8L+9)$. Thus, $\phi_{2}\in\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 8L+9)$. By Equation (3.2) and (3.4), we have

[TABLE]

for any ${\bm{\beta}}\in\{0,1,\cdots,K-1\}^{d}$. Equation (3.3) and $\phi_{2}=\psi_{2}\circ\psi_{1}$ implies

[TABLE]

Step $4\colon$ Construct the final network to implement the desired function $\phi$.

Define $\phi\coloneqq\phi_{2}\circ{\bm{\Phi}}_{1}+f({\bm{0}})-\omega_{f}(\sqrt{d})$. Since $\phi_{1}\in\mathcal{NN}(\textnormal{width}\leq 4\lfloor N^{1/d}\rfloor+3;\ \textnormal{depth}\leq 4L+5])$, we have ${\bm{\Phi}}_{1}\in\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width}\leq 4d\lfloor N^{1/d}\rfloor+3d;\ \textnormal{depth}\leq 4L+5;\ \textnormal{\#output}=d)$. Note that $\phi_{2}\in\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 8L+9)$. Thus, $\phi=\phi_{2}\circ{\bm{\Phi}}_{1}+f({\bm{0}})-\omega_{f}(\sqrt{d})$ is in

[TABLE]

Now let us estimate the approximation error. Note that $f={\widetilde{f}}+f({\bm{0}})-\omega_{f}(\sqrt{d})$. By Equation (3.5), for any ${\bm{x}}\in Q_{\bm{\beta}}$ and ${\bm{\beta}}\in\{0,1,\cdots,K-1\}^{d}$, we have

[TABLE]

where the last inequality comes from the fact $K=\lfloor N^{1/d}\rfloor^{2}\lfloor L^{2/d}\rfloor\geq\tfrac{N^{2/d}L^{2/d}}{8}$ for any $N,L\in\mathbb{N}^{+}$. Recall the fact $\omega_{f}(nr)\leq n\omega_{f}(r)$ for any $n\in\mathbb{N}^{+}$ and $r\in[0,\infty)$. Therefore, for any {\bm{x}}\in\cup_{\bm{\beta}\in\{0,1,\cdots,K-1\}^{d}}Q_{\bm{\beta}}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}=}[0,1]^{d}\backslash\Omega([0,1]^{d},K,\delta), we have

[TABLE]

It remains to show the upper bound of $\phi$. By Equation (3.6) and $\phi=\phi_{2}\circ{\bm{\Phi}}_{1}+f({\bm{0}})-\omega_{f}(\sqrt{d})$, it holds that $\|\phi\|_{L^{\infty}(\mathbb{R}^{d})}\leq|f({\bm{0}})|+\omega_{f}(\sqrt{d})$. Thus, we finish the proof.

3.3 Proof of Proposition 3.1

Lemma 3.3**.**

For any $N_{1},N_{2}\in\mathbb{N}^{+}$, given $N_{1}(N_{2}+1)+1$ samples $(x_{i},y_{i})\in\mathbb{R}^{2}$ with $x_{0}<x_{1}<\cdots<x_{N_{1}(N_{2}+1)}$ and $y_{i}\geq 0$ for $i=0,1,\cdots,N_{1}(N_{2}+1)$, there exists $\phi\in\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width\hskip 0.5ptvec}=[2N_{1},2N_{2}+1];\ \textnormal{\#output}=1)$ satisfying the following conditions.

1. (i)

$\phi(x_{i})=y_{i}$* for $i=0,1,\cdots,N_{1}(N_{2}+1)$;* 2. (ii)

$\phi$* is linear on each interval $[x_{i-1},x_{i}]$ for $i\notin\{(N_{2}+1)j:j=1,2,\cdots,N_{1}\}$.*

In fact, Lemma 3.3 is a part of Lemma $2.2$ in [61]. For the purpose of being self-contained, we present it as follows.

Lemma** (Lemma 2.2 of [61]).**

For any $m,n\in\mathbb{N}^{+}$, given any $m(n+1)+1$ samples $(x_{i},y_{i})\in\mathbb{R}^{2}$ with $x_{0}<x_{1}<x_{2}<\cdots<x_{m(n+1)}$ and $y_{i}\geq 0$ for $i=0,1,\cdots,m(n+1)$, there exists $\phi\in\mathcal{NN}(\textnormal{\#input}=1;\textnormal{width\hskip 0.5ptvec}=[2m,2n+1];\ \textnormal{\#output}=1)$ satisfying the following conditions.

1. (i)

$\phi(x_{i})=y_{i}$* for $i=0,1,\cdots,m(n+1)$;* 2. (ii)

$\phi$* is linear on each interval $[x_{i-1},x_{i}]$ for $i\notin\{(n+1)j:j=1,2,\cdots,m\}$;* 3. (iii)

$\displaystyle\sup_{x\in[x_{0},\,x_{m(n+1)}]}|\phi(x)|\leq 3\max_{i\in\{0,1,\cdots,m(n+1)\}}y_{i}\prod_{k=1}^{n}\left(1+\tfrac{\max\{x_{j(n+1)+n}-x_{j(n+1)+k-1}:j=0,1,\cdots,m-1\}}{\min\{x_{j(n+1)+k}-x_{j(n+1)+k-1}:j=0,1,\cdots,m-1\}}\right)$.

Lemma 3.4**.**

Given any $N,L,d\in\mathbb{N}^{+}$, it holds that

[TABLE]

Proof.

The key idea to prove Proposition 3.4 is to re-assemble ${\mathcal{O}}(L)$ sub-FNNs in the shallower FNN in the left of Figure 8 to form a deeper one with width ${\mathcal{O}}(N)$ and depth ${\mathcal{O}}(L)$ on the right of Figure 8.

For any $\phi\in\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width\hskip 0.5ptvec}=[N,NL];\ \textnormal{\#output}=1)$, $\phi$ can be implemented by a ReLU FNN described as

[TABLE]

where $\bm{g}$ and $\bm{h}$ are the output of the first hidden layer and the second hidden layer, respectively. Note that

[TABLE]

We can evenly divide $\bm{h}\in\mathbb{R}^{NL\times 1}$, $\bm{b}_{1}\in\mathbb{R}^{NL\times 1}$, $\bm{W}_{1}\in\mathbb{R}^{NL\times N}$, and $\bm{W}_{2}\in\mathbb{R}^{1\times NL}$ into $L$ parts as follows:

[TABLE]

and ${\bm{W}}_{2}=[\bm{W}_{2,1},\bm{W}_{2,2},\cdots,\bm{W}_{2,L}]$, where $\bm{h}_{\ell}\in\mathbb{R}^{N\times 1}$, $\bm{b}_{1,\ell}\in\mathbb{R}^{N\times 1}$, $\bm{W}_{1,\ell}\in\mathbb{R}^{N\times N}$, and $\bm{W}_{2,\ell}\in\mathbb{R}^{1\times N}$ for $\ell=1,2,\cdots,L$. Then, for $\ell=1,2,\cdots,L$, we have

[TABLE]

Define

[TABLE]

Then $\phi({\bm{x}})={\bm{W}}_{2}\cdot{\bm{h}}+{\bm{b}}_{2}=s_{L}+{\bm{b}}_{2}$ and

[TABLE]

Hence, it is easy to check that $\phi$ can also be implemented by the deep network shown in Figure 9.

It is clear that the network has the architecture of Figure 9 is with width $2N+2$ and depth $L+1$. So, we finish the proof. ∎

With Lemma 3.3 and 3.4 in hand, we are ready to present the detailed proof of Proposition 3.1.

Proof of Proposition 3.1.

We divide the proof into two cases: $d=1$ and $d\geq 2$.

Case $1\colon$ $d=1$.

In this case, $K=\lfloor N^{1/d}\rfloor^{2}\lfloor L^{2/d}\rfloor=N^{2}L^{2}$. Denote $M=N^{2}L$ and consider the sample set

[TABLE]

Its size is 2M+1=N\cdot\big{(}(2NL-1)+1\big{)}+1. By Lemma 3.3 (set $N_{1}=N$ and $N_{2}=2NL-1$ therein), there exists $\phi_{1}\in\mathcal{NN}(\textnormal{width\hskip 0.5ptvec}=[2N,2(2NL-1)+1])=\mathcal{NN}(\textnormal{width\hskip 0.5ptvec}=[2N,4NL-1])$ such that

• •

$\phi_{1}(\tfrac{M-1}{M})=\phi_{1}(1)=M-1$ and $\phi_{1}(\tfrac{m}{M})=\phi_{1}(\tfrac{m+1}{M}-\delta)=m$ for $m=0,1,\cdots,M-2$;

• •

$\phi_{1}$ is linear on $[\tfrac{M-1}{M},1]$ and each interval $[\tfrac{m}{M},\tfrac{m+1}{M}-\delta]$ for $m=0,1,\cdots,M-2$.

Then

[TABLE]

Now consider the another sample set

[TABLE]

Its size is 2L+1=1\cdot\big{(}(2L-1)+1\big{)}+1. By Lemma 3.3 (set $N_{1}=1$ and $N_{2}=2L-1$ therein), there exists $\phi_{2}\in\mathcal{NN}(\textnormal{width\hskip 0.5ptvec}=[2,2(2L-1)+1])=\mathcal{NN}(\textnormal{width\hskip 0.5ptvec}=[2,4L-1])$ such that

• •

$\phi_{2}(\tfrac{L-1}{ML})=\phi_{2}(\tfrac{1}{M})=L-1$ and $\phi_{2}(\tfrac{\ell}{ML})=\phi_{2}(\tfrac{\ell+1}{ML}-\delta)=\ell$ for $\ell=0,1,\cdots,L-2$;

• •

$\phi_{2}$ is linear on $[\tfrac{L-1}{ML},\tfrac{1}{M}]$ and each interval $[\tfrac{\ell}{ML},\tfrac{\ell+1}{ML}-\delta]$ for $\ell=0,1,\cdots,L-2$.

It follows that, for $m=0,1,\cdots,M-1$ and $\ell=0,1,\cdots,L-1$,

[TABLE]

The fact $K=ML$ implies each $k\in\{0,1,\cdots,K-1\}$ can be unique represented by $k=mL+\ell$ for $m=0,1,\cdots,M-1$ and $\ell=0,1,\cdots,L-1$. Then the desired function $\phi$ can be implemented by a ReLU FNN shown in Figure 10. Clearly,

[TABLE]

By Lemma 3.4, $\phi_{1}\in\mathcal{NN}(\textnormal{width\hskip 0.5ptvec}=[2N,4NL-1])\subseteq\mathcal{NN}(\textnormal{width}\leq 4N+2;\ \textnormal{depth}\leq 2L+1)$ and $\phi_{2}\in\mathcal{NN}(\textnormal{width\hskip 0.5ptvec}=[2,4L-1])\subseteq\mathcal{NN}(\textnormal{width}\leq 6;\ \textnormal{depth}\leq 2L+1)$, implying $\phi\in\mathcal{NN}(\textnormal{width}\leq\max\{4N+2+1,6+1\}=4N+3;\ \textnormal{depth}\leq(2L+1)+2+(2L+1)+1=4L+5)$. So we finish the proof for the case $d=1$.

Case $2\colon$ $d\geq 2$.

Now we consider the case when $d\geq 2$. Consider the sample set

[TABLE]

whose size is 2K+1=\lfloor N^{1/d}\rfloor\big{(}(2\lfloor N^{1/d}\rfloor\lfloor L^{2/d}\rfloor-1)+1\big{)}+1. By Lemma 3.3 (set $N_{1}=\lfloor N^{1/d}\rfloor$ and $N_{2}=2\lfloor N^{1/d}\rfloor\lfloor L^{2/d}\rfloor-1$ therein), there exists $\phi$ in

[TABLE]

such that

• •

$\phi(\tfrac{K-1}{K})=\phi(1)=K-1$, and $\phi(\tfrac{k}{K})=\phi(\tfrac{k+1}{K}-\delta)=k$ for $k=0,1,\cdots,K-2$;

• •

$\phi$ is linear on $[\tfrac{K-1}{K},1]$ and each interval $[\tfrac{k}{K},\tfrac{k+1}{K}-\delta]$ for $k=0,1,\cdots,K-2$.

Then

[TABLE]

By Lemma 3.4,

[TABLE]

which means we finish the proof for the case $d\geq 2$. ∎

3.4 Proof of Proposition 3.2

The proof of Proposition 3.2 is based on the bit extraction technique in [5, 27]. In fact, we modify this technique to extract the sum of many bits rather than one bit and this modification can be summarized in Lemma 3.5 and 3.6 below.

Lemma 3.5**.**

For any $L\in\mathbb{N}^{+}$, there exists a function $\phi$ in

[TABLE]

such that, for any $\theta_{1},\theta_{2},\cdots,\theta_{L}\in\{0,1\}$, we have

[TABLE]

Proof.

Given $\theta_{1},\theta_{2},\cdots,\theta_{L}\in\{0,1\}$, define

[TABLE]

and

[TABLE]

Then we have

[TABLE]

and

[TABLE]

I would like to point out that, by above two iteration equations, we can iteratively get $\xi_{1},\theta_{1},\xi_{2},\theta_{2},\cdots,\xi_{L},\theta_{L}$ when $\xi_{1}$ is given. Based on this iteration idea, the rest proof can be divided into three steps.

Step $1\colon$ Simplify the iteration equations.

Note that $\mathcal{T}(x)=\sigma(x/\delta+1)-\sigma(x/\delta)$ for any $x\notin(-\delta,0)$. By setting $\delta=1/2-\sum_{j=2}^{L}2^{-j}=2^{-L}$, we have $\xi_{j}-1/2\notin(-\delta,0)$ for all $j$, implying

[TABLE]

for $j=1,2,\cdots,L$, where $\mathcal{L}$ is the linear map given by $\mathcal{L}(x)=(x-1/2)/\delta$. It follows that, for $j=1,2,\cdots,L-1$,

[TABLE]

Step $2\colon$ Design a ReLU FNN to output $\sum_{j=1}^{\ell}\theta_{j}$.

It is easy to design a ReLU FNN to output $\theta_{1},\theta_{2},\cdots,\theta_{L}$ by Equation (3.11) and (3.12) when using $\xi_{1}=\textnormal{bin}\hskip 1.2pt0.\theta_{1}\theta_{2}\cdots\theta_{L}$ as the input. However, it is highly non-trivial to construct a ReLU FNN to output $\sum_{j=1}^{\ell}\theta_{j}$ with another input $\ell$, since many operations like multiplication and comparison are not allowed in designing ReLU FNNs.

Now let us establish a formula to represent $\sum_{j=1}^{\ell}\theta_{j}$ in a form of a ReLU FNN as follows:

The fact that $x_{1}x_{2}=\sigma(x_{1}+x_{2}-1)$ for any $x_{1},x_{2}\in\{0,1\}$ implies

[TABLE]

for $\ell=1,2,\cdots,L$, where the last equality comes from the fact $\mathcal{T}(n)=\sigma(n+1)-\sigma(n)$ for any integer $n$.

To simplify the notations, we define

[TABLE]

for $\ell=1,2,\cdots,L$ and $j=1,2,\cdots,L$. Then,

[TABLE]

With Equation (3.11), (3.12), (3.13), and (3.14) in hand, it is easy to construct a function $\phi$ implemented by a ReLU FNN with the desired width and depth outputting $\sum_{j=1}^{\ell}\theta_{j}=\sum_{j=1}^{L}z_{\ell,j}$ given the input $(\xi_{1},\ell)=(\textnormal{bin}\hskip 1.2pt0.\theta_{1}\theta_{2}\cdots\theta_{L},\ell)$ for $\ell\in\{1,2,\cdots,L\}$ and $\theta_{1},\theta_{2},\cdots,\theta_{L}\in\{0,1\}$. The details of construction are shown in Figure 11. Clearly, the network in Figure 11 is with width $7$ and depth $2L+1$, which implies

[TABLE]

So we finish the proof. ∎

Next, we introduce Lemma 3.6 as an advanced version of Lemma 3.5.

Lemma 3.6**.**

For any $N,L\in\mathbb{N}^{+}$, any $\theta_{m,\ell}\in\{0,1\}$ for $m=0,1,\cdots,M-1$ and $\ell=0,1,\cdots,L-1$, where $M=N^{2}L$, there exists a function $\phi$ implemented by a ReLU FNN with width $4N+3$ and depth $3L+3$ such that

[TABLE]

Proof.

Define

[TABLE]

Consider the sample set $\{(m,y_{m}):m=0,1,\cdots,M\}$, whose cardinality is M+1=N\big{(}(NL-1)+1\big{)}+1. By Lemma 3.3 (set $N_{1}=N$ and $N_{2}=NL-1$ therein), there exists

[TABLE]

such that

[TABLE]

By Lemma 3.5, there exists

[TABLE]

such that, for any $\xi_{1},\xi_{2},\cdots,\xi_{L}\in\{0,1\}$, we have

[TABLE]

It follows that, for any $\xi_{0},\xi_{1},\cdots,\xi_{L-1}\in\{0,1\}$, we have

[TABLE]

Thus, for $m=0,1,\cdots,M-1$ and $\ell=0,1,\cdots,L-1$, we have

[TABLE]

Hence, the desired function function $\phi$ can be implemented by the network shown in Figure 12. By Lemma 3.4, $\phi_{1}\in\mathcal{NN}(\textnormal{width\hskip 0.5ptvec}=[2N,2NL-1])\subseteq\mathcal{NN}(\textnormal{width}\leq 4N+2;\ \textnormal{depth}\leq L+1)$, implying the network in Figure 12 is with width $\max\{(4N+2)+1,7\}=4N+3$ and depth $(2L+1)+1+(L+1)=3L+3$.

So we finish the proof. ∎

Next, we apply Lemma 3.6 to prove Lemma 3.7 below, which is a key intermediate conclusion to prove Proposition 3.2.

Lemma 3.7**.**

For any $\varepsilon>0$, $L,N\in\mathbb{N}^{+}$, denote $M=N^{2}L$ and assume $\{y_{m,\ell}\geq 0:m=0,1,\cdots,M-1\textnormal{ and }\ell=0,1,\cdots,L-1\}$ is a sample set with

[TABLE]

Then there exists $\phi\in\mathcal{NN}(\textnormal{\#input}=2;\ \textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 3L+6)$ such that

1. (i)

$|\phi(m,\ell)-y_{m,\ell}|\leq\varepsilon$, for $m=0,1,\cdots,M-1\textnormal{ and }\ell=0,1,\cdots,L-1$; 2. (ii)

$0\leq\phi(x_{1},x_{2})\leq\max\{y_{m,\ell}:m=0,1,\cdots,M-1\textnormal{ and }\ell=0,1,\cdots,L-1\}$, for any $x_{1},x_{2}\in\mathbb{R}$.

Proof.

Define

[TABLE]

We will construct a function implemented by a ReLU FNN to map the index $(m,\ell)$ to $a_{m,\ell}\varepsilon$ for $m=0,1,\cdots,M-1\textnormal{ and }\ell=0,1,\cdots,L-1$.

Define $b_{m,0}\coloneqq 0$ and $b_{m,\ell}\coloneqq a_{m,\ell}-a_{m,\ell-1}$ for $m=0,1,\cdots,M-1\textnormal{ and }\ell=1,\cdots,L-1$. Since $|y_{m,\ell}-y_{m,\ell-1}|\leq\varepsilon$ for all $m$ and $\ell$, we have $b_{m,\ell}\in\{-1,0,1\}$. Hence, there exist $c_{m,\ell}$ and $d_{m,\ell}\in\{0,1\}$ such that $b_{m,\ell}=c_{m,\ell}-d_{m,\ell}$, which implies

[TABLE]

for $m=0,1,\cdots,M-1\textnormal{ and }\ell=1,\cdots,L-1$.

For the sample set $\{(m,a_{m,0}):m=0,1,\cdots,M-1\}\cup\{(M,0)\}$, whose size is M+1=N\cdot\big{(}(NL-1)+1\big{)}+1, by Lemma 3.3 (set $N_{1}=N$ and $N_{2}=NL-1$ therein), there exists $\psi_{1}\in\mathcal{NN}(\textnormal{width\hskip 0.5ptvec}=[2N,2(NL-1)+1])=\mathcal{NN}(\textnormal{width\hskip 0.5ptvec}=[2N,2NL-1])$ such that

[TABLE]

By Lemma 3.6, there exist $\psi_{2},\psi_{3}\in\mathcal{NN}(\textnormal{width}\leq 4N+3;\ \textnormal{depth}\leq 3L+3)$ such that

[TABLE]

for $m=0,1,\cdots,M-1\textnormal{ and }\ell=0,1,\cdots,L-1$. Hence, it holds that

[TABLE]

for $m=0,1,\cdots,M-1\textnormal{ and }\ell=0,1,\cdots,L-1$.

Define

[TABLE]

Then the desired function can be implemented by two sub-networks shown in Figure 13.

By Lemma 3.4, $\psi_{1}\in\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width\hskip 0.5ptvec}=[2N,2NL-1])\subseteq\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width}\leq 4N+2;\textnormal{depth}\leq L+1)$. Note that $\psi_{2},\psi_{3}\in\mathcal{NN}(\textnormal{width}\leq 4N+3;\ \textnormal{depth}\leq 3L+3)$. Thus, $\phi_{1}\in\mathcal{NN}(\textnormal{width}\leq(4N+2)+2(4N+3)=12N+8;\ \textnormal{depth}\leq(3L+3)+1=3L+4)$ as shown in Figure 13. And it is clear that $\phi_{2}\in\mathcal{NN}(\textnormal{width}\leq 4;\ \textnormal{depth}\leq 2)$, implying $\phi=\phi_{2}\circ\phi_{1}\in\mathcal{NN}(\textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq(3L+4)+2=3L+6)$.

Clearly, $0\leq\phi(x_{1},x_{2})\leq y_{\textnormal{max}}$ for any $x_{1},x_{2}\in\mathbb{R}$, since $\phi(x_{1},x_{2})=\phi_{2}\circ\phi_{1}(x_{1},x_{2})=\max\{\sigma(\phi_{1}(x_{1},x_{2})),y_{\textnormal{max}}\}$.

Note that $0\leq\varepsilon a_{m,\ell}=\varepsilon\lfloor y_{m,\ell}/\varepsilon\rfloor\leq y_{\textnormal{max}}$. Then we have $\phi(m,\ell)=\phi_{2}\circ\phi_{1}(m,\ell)=\phi_{2}(\varepsilon a_{m,\ell})=\max\{\sigma(\varepsilon a_{m,\ell}),y_{\textnormal{max}}\}=\varepsilon a_{m,\ell}$. Therefore,

[TABLE]

for $m=0,1,\cdots,M-1$ and $\ell=0,1,\cdots,L-1$. Hence, we finish the proof. ∎

Finally, we apply Lemma 3.7 to prove Proposition 3.2.

Proof of Proposition 3.2.

Let $M=N^{2}L$, then we may assume $J=ML$ since we can set $y_{J-1}=y_{J}=y_{J+1}=\cdots=y_{ML-1}$ if $J<ML$.

For the sample set

[TABLE]

whose size is 2M+1=N\cdot\big{(}(2NL-1)+1\big{)}+1, by Lemma 3.3 (set $N_{1}=N$ and $N_{2}=NL-1$ therein), there exist $\phi_{1}\in\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width\hskip 0.5ptvec}=[2N,2(2NL-1)+1])=\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width\hskip 0.5ptvec}=[2N,4NL-1])$ such that

• •

$\phi_{1}(ML)=M$ and $\phi_{1}(mL)=\phi_{1}(mL+L-1)=m$ for $m=0,1,\cdots,M-1$;

• •

$\phi_{1}$ is linear on each interval $[mL,mL+L-1]$ for $m=0,1,\cdots,M-1$.

It follows that

[TABLE]

for $m=0,1,\cdots,M-1$ and $\ell=0,1,\cdots,L-1$.

Note that any number $j$ in $\{0,1,\dots,J-1\}$ can be uniquely indexed as $j=mL+\ell$ for $m=0,1,\cdots,M-1$ and $\ell=0,1,\cdots,L-1$. So we can denote $y_{j}=y_{mL+\ell}$ as $y_{m,\ell}$. Then by Lemma 3.7, there exists $\phi_{2}\in\mathcal{NN}(\textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 3L+6)$ such that

[TABLE]

and

[TABLE]

where $y_{\textnormal{max}}\coloneqq\max\{y_{m,\ell}:m=0,1,\cdots,M-1\textnormal{ and }\ell=0,1,\cdots,L-1\}=\max\{y_{j}:j=0,1,\cdots,ML-1\}$.

Note that $\phi_{1}\in\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width\hskip 0.5ptvec}=[2N,4NL-1])\subseteq\mathcal{NN}(\textnormal{\#input}=1;\ \textnormal{width}\leq 8N+2;\textnormal{depth}\leq L+1)$ by Lemma 3.4 and $\phi_{2}\in\mathcal{NN}(\textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq 3L+6)$. So $\phi\in\mathcal{NN}(\textnormal{width}\leq 12N+8;\ \textnormal{depth}\leq(L+1)+2+(3L+6)=4L+9)$ as shown in Figure 14.

Equation (3.18) implies

[TABLE]

since $\phi$ is given by \phi(x)=\phi_{2}\big{(}\phi_{1}(x),x-L\phi_{1}(x)\big{)}.

Represent $j\in\{0,1,\cdots,ML-1\}$ via $j=mL+\ell$ for $m=0,1,\cdots,M-1$ and $\ell=0,1,\cdots,L-1$, then we have, by Equation (3.17),

[TABLE]

So we finish the proof. ∎

We would like to remark that the key idea in the proof of Proposition 3.2 is the bit extraction technique in Lemma 3.5, which allows us to store $L$ bits in a binary number $\textnormal{bin}\hskip 1.2pt0.\theta_{1}\theta_{2}\cdots\theta_{L}$ and extract each bit $\theta_{i}$. The extraction operator can be efficiently carried out via a deep ReLU neural network demonstrating the power of depth.

4 Neural networks approximation and evaluation in practice

This section is concerned with neural networks approximation and evaluation in practice, e.g., approximating functions defined on irregular domains or domains with a low-dimensional structure, and neural network computation in parallel computing. In the practical training of FNNs, the approximation rate in this paper can only be observed if the global minimizers of neural network optimization can be identified. Since there is no existing optimization algorithm guaranteeing a global minimizer, it is challenging to observe the proved approximation rate currently. Developing optimization algorithms for global minimizers is another interesting research topic as a future work.

4.1 Approximation on irregular domain

In this section, we consider approximating continuous functions defined on irregular domains by deep ReLU FNNs. The construction is through extending the target function to a cubic domain, applying Theorem 1.1, and finally restricting the constructed FNN back to the irregular domain.

Given any uniformly continuous and real-valued function $f$ defined on a metric space $S$ with a metric $d_{S}(\cdot,\cdot)$, we define the (optimal) modulus of continuity of $f$ on a subset $E\subseteq S$ as

[TABLE]

For the purpose of consistency and simplicity, $\omega_{f}(\cdot)$ is short of $\omega_{f}^{[0,1]^{d}}(\cdot)$.

First, let us present two lemmas for (approximately) extending (almost) continuous functions on $E$ to (almost) continuous functions on $S$. These lemmas are similar to the well-known results for extending Lipschitz or differentiable functions in [46, 63]. We generalize these results to a broader class of functions required in the proof of Theorem 4.3.

Lemma 4.1** (Approximate Extension of Almost-Continuous Functions).**

Assume $S$ is a metric space with a metric $d_{S}(\cdot,\cdot)$ and $\omega:[0,\infty)\to[0,\infty)$ is an increasing function with

[TABLE]

Let $f$ be a real-valued function defined on a subset $E\subseteq S$ and satisfy

[TABLE]

where $\Delta$ is a positive constant independent of $f$. Then there exists a function $g$ defined on $S$ such that

[TABLE]

and

[TABLE]

In Lemma 4.1, $g$ is an approximate extension of $f$ defined on $E$ to a new domain $S$ with an approximation error $\omega(\Delta)$. In a special case when $\Delta=0$ and $\omega(0)=0$, $g$ is an exact extension of $f$.

Proof of Lemma 4.1.

Define

[TABLE]

By Equation (4.2), we have $f({\bm{x}}_{1})-\omega(d_{S}({\bm{x}}_{1},{\bm{x}}_{2})+\Delta)\leq f({\bm{x}}_{2})$ for any ${\bm{x}}_{1},{\bm{x}}_{2}\in E$. It holds that $g({\bm{x}})\leq f({\bm{x}})$ for any ${\bm{x}}\in E$. Together with

[TABLE]

for any ${\bm{x}}\in E$, it follows that $0\leq f({\bm{x}})-g({\bm{x}})\leq\omega(\Delta)$ for any ${\bm{x}}\in E$. By Equation (4.1) and the fact

[TABLE]

we have

[TABLE]

for any ${\bm{x}}_{1},{\bm{x}}_{2}\in S$. Similarly, we have $g({\bm{x}}_{2})-g({\bm{x}}_{1})\leq\omega(d_{S}({\bm{x}}_{1},{\bm{x}}_{2})),$ which implies

[TABLE]

So we finish the proof. ∎

Next, we introduce a lemma below for extending continuous functions defined on $E\subseteq S$ to continuous functions defined on $S$ preserving the modulus of continuity.

Lemma 4.2** (Extension of Continuous Functions).**

Suppose $f$ is a uniformly continuous function defined on a subset $E\subseteq S$, where $S$ is a metric space with a metric $d_{S}(\cdot,\cdot)$, then there exists a uniformly continuous function $g$ on $S$ such that $f({\bm{x}})=g({\bm{x}})$ for ${\bm{x}}\in E$ and $\omega_{f}^{E}(r)=\omega_{g}^{S}(r)$ for any $r\geq 0$.

Proof.

By the application of Lemma 4.1 with $\omega(r)=\omega_{f}^{E}(r)$ for $r\geq 0$ and $\Delta=0$, we know that there exists $g:S\to\mathbb{R}$ such that

[TABLE]

and

[TABLE]

The equation above and the uniform continuity of $f$ imply that $g$ is uniformly continuous. It also follows that

[TABLE]

since $\omega_{g}^{S}(\cdot)$ is the optimal modulus of continuity of $g$. Note that $\omega_{f}^{E}(\cdot)$ is the optimal moduls of continuity of $f$ and

[TABLE]

Hence, $\omega_{f}^{E}(r)\leq\omega_{g}^{S}(r)$ for all $r\geq 0$, which implies $\omega_{f}^{E}(r)=\omega_{g}^{S}(r)$ since we have proved that $\omega_{g}^{S}(r)\leq\omega_{f}^{E}(r)$ for all $r\geq 0$. So we finish the proof. ∎

Now we are ready to introduce and prove the main theorem of this section, which extends Theorem 1.1 to an irregular domain as follows.

Theorem 4.3**.**

Let $f$ be a uniformly continuous function defined on $E\subseteq[-R,R]^{d}$. For arbitrary $L\in\mathbb{N}^{+}$ and $N\in\mathbb{N}^{+}$, there exists a function $\phi$ implemented by a ReLU FNN with width 3^{d+3}\max\big{\{}d\lfloor N^{1/d}\rfloor,\,N+1\big{\}} and depth $12L+14+2d$ such that

[TABLE]

Proof.

By Lemma 4.2, $f$ can be extended to $\mathbb{R}^{d}$ such that

[TABLE]

Define

[TABLE]

It follows that

[TABLE]

By Theorem 1.1, there exists a function ${\widetilde{\phi}}$ implemented by a ReLU FNN with width 3^{d+3}\max\big{\{}d\lfloor N^{1/d}\rfloor,\,N+1\big{\}} and depth $12L+14+2d$ such that

[TABLE]

Define

[TABLE]

Then, by Equation (4.3), for any ${\bm{x}}\in E\subseteq[-R,R]^{d}$, we have

[TABLE]

which implies

[TABLE]

So we finish the proof. ∎

4.2 Approximation in a neighborhood of a low-dimensional manifold

In this section, we study neural network approximation of functions defined in a neighborhood of a low-dimensional manifold and prove Theorem 1.2 in this setting. Let us first introduce Theorem 4.4 which is required to prove Theorem 1.2.

Theorem 4.4** (Theorem $3.1$ of [3]).**

Let $\mathcal{M}$ be a compact $d_{\mathcal{M}}$-dimensional Riemannian submanifold of $\mathbb{R}^{d}$ having condition number $1/\tau$, volume $V$, and geodesic covering regularity $\mathcal{R}$. Fix $\delta\in(0,1)$ and $\gamma\in(0,1)$. Let $\bm{A}=\sqrt{\tfrac{d}{d_{\delta}}}\Phi$, where $\Phi\in\mathbb{R}^{d_{\delta}\times d}$ is a random orthoprojector with

[TABLE]

If $d_{\delta}\leq d$, then with probability at least $1-\gamma$, the following statement holds: For every ${\bm{x}}_{1},{\bm{x}}_{2}\in\mathcal{M}$,

[TABLE]

Theorem 4.4 shows the existence of a linear projector $\bm{A}\in\mathbb{R}^{d_{\delta}\times d}$ that maps a low-dimensional manifold in a high-dimensional space to a low-dimensional space nearly preserving distance. With this projection $\bm{A}$ available, we can prove Theorem 1.2 via constructing a ReLU FNN defined in the low-dimensional space using Theorem 4.3 and hence the curse of dimensionality is lessened. The ideas of the proof are summarized in the following Table 1.

In Table 1 and the detailed proof later, we introduce a new notation ${\mathcal{SL}}(E)$ for any compact set $E\subseteq\mathbb{R}^{d}$ as the “smallest” element of $E$. Specifically, ${\mathcal{SL}}(E)$ is defined as the unique point in $\cap_{k=1}^{d}E_{k}$, where

[TABLE]

and $E_{0}=E$. The compactness of $E$ ensures that $\cap_{k=1}^{d}E_{k}$ is in fact one point belonging to $E$. The introduction of ${\mathcal{SL}}(\cdot)$ uniquely formulates a low-dimensional function $\widetilde{f}$ representing a high-dimensional function $f$ defined on $\mathcal{M}_{\varepsilon}$ by

[TABLE]

As we shall see later, such a definition of ${\widetilde{f}}$ is reasonable because $\{{\bm{x}}\in\mathcal{M}_{\varepsilon}:\bm{A}{\bm{x}}={\bm{y}}\}$ is contained in a small ball of radius ${\mathcal{O}}(\varepsilon)$ for any ${\bm{y}}\in\bm{A}(\mathcal{M}_{\varepsilon})$. There are many other alternative ways to define ${\mathcal{SL}}(\cdot)$ as long as the definition ensures that ${\mathcal{SL}}(E)$ contains only one element. For example, ${\mathcal{SL}}(E)$ can be defined as any arbitrary point in $E$. For another example, ${\bm{y}}\in\bm{A}(\mathcal{M})$ cannot guarantee {\bm{x}}_{\bm{y}}={\mathcal{SL}}\big{(}\{{\bm{x}}\in\mathcal{M}_{\varepsilon}:\bm{A}{\bm{x}}={\bm{y}}\}\big{)}\in\mathcal{M} in the current definition, but in practice we can choose {\mathcal{SL}}\big{(}\{{\bm{x}}\in\mathcal{M}:\bm{A}{\bm{x}}={\bm{y}}\}\big{)} as ${\bm{x}}_{\bm{y}}$ to ensure that ${\bm{x}}_{\bm{y}}\in\mathcal{M}$, which might be beneficial for potential applications.

Now we are ready to prove Theorem 1.2.

Proof of Theorem 1.2.

By Theorem 4.4, there exists a matrix $\bm{A}\in\mathbb{R}^{d_{\delta}\times d}$ such that

[TABLE]

where $\bm{I}_{d_{\delta}}$ is an identity matrix of size $d_{\delta}\times d_{\delta}$, and

[TABLE]

Given any ${\bm{y}}\in\bm{A}(\mathcal{M}_{\varepsilon})$, then $\{{\bm{x}}\in\mathcal{M}_{\varepsilon}:\bm{A}{\bm{x}}={\bm{y}}\}$ is a nonzero compact set. Let {\bm{x}}_{\bm{y}}={\mathcal{SL}}\big{(}\{{\bm{x}}\in\mathcal{M}_{\varepsilon}:\bm{A}{\bm{x}}={\bm{y}}\}\big{)}, then we define ${\widetilde{f}}$ on $\bm{A}(\mathcal{M}_{\varepsilon})$ as ${\widetilde{f}}({\bm{y}})=f({\bm{x}}_{\bm{y}}).$

For any ${\bm{y}}_{1},{\bm{y}}_{2}\in\bm{A}(\mathcal{M}_{\varepsilon})$, let {\bm{x}}_{i}={\mathcal{SL}}\big{(}\{{\bm{x}}\in\mathcal{M}_{\varepsilon}:\bm{A}{\bm{x}}={\bm{y}}_{i}\}\big{)}, then ${\bm{x}}_{i}\in\mathcal{M}_{\varepsilon}$ for $i=1,2$. By the definition of $\mathcal{M}_{\varepsilon}$, there exist ${\widetilde{\bm{x}}}_{1},{\widetilde{\bm{x}}}_{2}\in\mathcal{M}$ such that $|{\widetilde{\bm{x}}}_{i}-{\bm{x}}_{i}|\leq\varepsilon$ for $i=1,2$. It follows that

[TABLE]

where the last inequality comes from Equation (4.5). By the triangular inequality, we have

[TABLE]

Set $\omega(r)=\omega_{f}(\tfrac{1}{1-\delta}r)$ for any $r\geq 0$ and $\Delta=2\varepsilon\sqrt{\tfrac{d}{d_{\delta}}}+2\varepsilon(1-\delta)$, then

[TABLE]

By Lemma 4.1, there exists ${\widetilde{g}}$ defined on $\mathbb{R}^{d_{\delta}}$ such that

[TABLE]

and

[TABLE]

It follows that

[TABLE]

By Equation (4.4) and the definition of $\mathcal{M}_{\varepsilon}$ in Equation (1.2), it is easy to check that

[TABLE]

By the application of Theorem 4.3 with $E=[-\sqrt{\tfrac{d}{d_{\delta}}},\sqrt{\tfrac{d}{d_{\delta}}}]^{d_{\delta}}$, there exists a function ${\widetilde{\phi}}$ implemented by a ReLU FNN with width 3^{d_{\delta}+3}\max\big{\{}d_{\delta}\lfloor N^{1/d_{\delta}}\rfloor,\,N+1\big{\}} and depth $12L+14+2d_{\delta}$ such that

[TABLE]

Define $\phi\coloneqq{\widetilde{\phi}}\circ\bm{A}$, i.e., $\phi({\bm{x}})\coloneqq{\widetilde{\phi}}(\bm{A}{\bm{x}})$ for any ${\bm{x}}\in\mathbb{R}^{d}$. Then $\phi$ is also a ReLU FNN with width 3^{d_{\delta}+3}\max\big{\{}d_{\delta}\lfloor N^{1/d_{\delta}}\rfloor,\,N+1\big{\}} and depth $12L+14+2d_{\delta}$.

For any ${\bm{x}}\in\mathcal{M}_{\varepsilon}$, set ${\bm{y}}=\bm{A}{\bm{x}}$ and {\bm{x}}_{\bm{y}}={\mathcal{SL}}\big{(}\{{\bm{z}}\in\mathbb{R}^{d}:\bm{A}{\bm{z}}={\bm{y}}\}\big{)}, there exist ${\widetilde{\bm{x}}},{\widetilde{\bm{x}}}_{\bm{y}}\in\mathcal{M}$ such that $|{\widetilde{\bm{x}}}-{\bm{x}}|\leq\varepsilon$ and $|{\widetilde{\bm{x}}}_{\bm{y}}-{\bm{x}}_{\bm{y}}|\leq\varepsilon$. It follows from Equation (4.5) that

[TABLE]

In fact, the above equation implies that $\{{\bm{x}}\in\mathcal{M}_{\varepsilon}:\bm{A}{\bm{x}}={\bm{y}}\}$ is contained in a small ball of radius ${\mathcal{O}}(\varepsilon)$ for ${\bm{y}}\in\bm{A}(\mathcal{M}_{\varepsilon})$ as we mentioned previously.

Together with Equation (4.6), (4.7), (4.8), and (4.9), we have, for any ${\bm{x}}\in\mathcal{M}_{\varepsilon}$,

[TABLE]

Hence, we have finished the proof of this theorem. ∎

It is worth emphasizing that the approximation error

[TABLE]

in Theorem 1.2 is equal to {\mathcal{O}}\Big{(}\omega_{f}\big{(}{\mathcal{O}}(N^{-2/d_{\delta}}L^{-2/d_{\delta}})\big{)}\Big{)} when $\varepsilon={\mathcal{O}}(N^{-2/d_{\delta}}L^{-2/d_{\delta}})$.

The application of Theorem 4.4 and the proof of Theorem 1.2 in fact inspire an efficient two-step algorithm for high-dimensional learning problems: in the first step, high-dimensional data are projected to a low-dimensional space via a random projection; in the second step, a deep learning algorithm is applied to learn from the low-dimensional data. By Theorem 4.4 and 1.2, the deep learning algorithm in the low-dimensional space can still provide good results with a high probability.

4.3 Optimal ReLU FNN structure in parallel computing

In this section, we show how to select the best ReLU FNN to approximate functions in {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}B_{\lambda}(C^{\alpha}([0,1]^{d}))} on a $d$-dimensional cube, if the approximation error $\varepsilon$ and the number of parallel computing cores (processors) $p$ are given. We choose the best ReLU FNN by minimizing the time complexity in each training iteration. The analysis in this section is valid up to a constant prefactor.

Assume $\phi_{\bm{\theta}}\in\mathcal{NN}(\textnormal{\#input}=d;\ \textnormal{width\hskip 0.5ptvec}=[N]^{L};\ \textnormal{\#output}=1)$, $N,L\in\mathbb{N}^{+}$, where $\bm{\theta}$ is the vector including all parameters of $\phi_{\bm{\theta}}$. By the basic knowledge of parallel computing (see [36] for more details), we have the following Table 2.

For the sake of simplicity, we assume that the training batch size is ${\mathcal{O}}(1)$. Denote the time complexity of each training iteration as $T(n,L)$, then

[TABLE]

Theorem 1.1 and 2.3 imply that the approximation error $\varepsilon$ is essentially ${\mathcal{O}}((NL)^{-2\alpha/d})$. Hence, we can get the optimal size of ReLU FNNs via the optimization problem below:

[TABLE]

To simplify the discussion, we have the following assumptions:

• •

Dropping the notation ${\mathcal{O}}(\cdot)$ sometimes while assuming asymptotic analysis with the abuse of notations.

• •

$N$, $L$, and $p$ are allowed to be real numbers.

• •

We denote $\varepsilon=(NL)^{-2\alpha/d}$ since the approximation rate {\mathcal{O}}\big{(}(NL)^{-2\alpha/d}\big{)} is both attainable and nearly optimal.

With $\varepsilon=(NL)^{-2\alpha/d}$, we have

[TABLE]

Then we get $T(N,L)=\mathcal{O}(\overline{T}(N,L))$. Therefore, the optimization problem in Equation (4.10) can be simplified to

[TABLE]

By Equation (4.11), $\overline{T}(N,L)$ is independent of $L$ on the condition that $\varepsilon=(NL)^{-2\alpha/d}$. Therefore, we may denote $\overline{T}(N,L)$ by $\overline{T}(N)$. Now we consider two cases: the case $p={\mathcal{O}}(1)$ and the case $p\gg{\mathcal{O}}(1)$.

Case $1\colon$ The case $p={\mathcal{O}}(1)$.

It is clear that $\overline{T}(N)$ is increasing in $N$ when $N\in[p,\infty)$ by Equation (4.11). Together with $p={\mathcal{O}}(1)$, then ${\mathcal{O}}(\sqrt{p})={\mathcal{O}}(p)={\mathcal{O}}(1)$. Therefore, $N_{\textnormal{opt}}={\mathcal{O}}(1)$ and $L_{\textnormal{opt}}={\mathcal{O}}(\varepsilon^{-d/(2\alpha)})$. Note that we regard $d$ as a constant (${\mathcal{O}}(1)$) in above analysis, $N_{\textnormal{opt}}$ should be ${\mathcal{O}}(d)$ in fact.

Case $2\colon$ The case $p\gg{\mathcal{O}}(1)$.

Since $\varepsilon=(NL)^{-2\alpha/d}$, we have $N\leq\varepsilon^{-d/(2\alpha)}$. We only need to consider the monotonicity of $\overline{T}(N)$ on $[1,\varepsilon^{-d/(2\alpha)}]$. Together with Equation (4.11), this case can be divided into two sub-cases: the sub-case $\sqrt{p}\leq\varepsilon^{-d/(2\alpha)}$ and the sub-case $\sqrt{p}>\varepsilon^{-d/(2\alpha)}$.

Case $2.1\colon$ The sub-case $\sqrt{p}>\varepsilon^{-d/(2\alpha)}$.

$\sqrt{p}>\varepsilon^{-d/(2\alpha)}$ implies $[1,\varepsilon^{-d/(2\alpha)}]\subseteq[1,\sqrt{p}]$. Hence, $\overline{T}(N)$ is decreasing in $N$ on $[1,\varepsilon^{-d/(2\alpha)}]$. It follows that $N_{\textnormal{opt}}={\mathcal{O}}(\varepsilon^{-d/(2\alpha)})$ and that $L_{\textnormal{opt}}={\mathcal{O}}(1)$.

Case $2.2\colon$ The sub-case $\sqrt{p}\leq\varepsilon^{-d/(2\alpha)}$.

For this sub-case, $N_{\textnormal{opt}}$ and $N_{\textnormal{opt}}$ are hard to estimate. However, we can give a rough range of $N_{\textnormal{opt}}$. Since $\overline{T}(N)$ is decreasing in $N$ on $[1,\sqrt{p}]$ and increasing in $N$ on $[p,\infty)$, the minimum of $\overline{T}(N)$ is achieved on $[\sqrt{p},p]$. Hence, $N_{\textnormal{opt}}\in[{\mathcal{O}}(\sqrt{p}),{\mathcal{O}}(p)]\cap[{\mathcal{O}}(\sqrt{p}),{\mathcal{O}}(\varepsilon^{-d/(2\alpha)})]$ and $L_{\textnormal{opt}}={\mathcal{O}}(\varepsilon^{-d/(2\alpha)}/N_{\textnormal{opt}})$.

5 Conclusion and future work

This paper aims at a quantitative and optimal approximation rate of ReLU FNNs in terms of both width and depth simultaneously to approximate continuous functions. It was shown that ReLU FNNs with width ${\mathcal{O}}(N)$ and depth ${\mathcal{O}}(L)$ can approximate an arbitrary continuous function on a $d$-dimensional cube with an approximation rate $19\sqrt{d}\,\omega_{f}(N^{-2/d}L^{-2/d})$. In particular, when $f$ is a Hölder continuous function of order $\alpha$ with a Hölder constant $\lambda$, the approximation rate is $19\sqrt{d}\,\lambda N^{-2\alpha/d}L^{-2\alpha/d}$ and it is nearly asymptotically tight. We also extended our analysis to the case when the domain of $f$ is irregular and showed the same approximation rate. In practical applications, it is usually believed that real data are sampled from an $\varepsilon$-neighborhood of a $d_{\mathcal{M}}$-dimensional smooth manifold $\mathcal{M}\subseteq[0,1]^{d}$ with $d_{\mathcal{M}}\ll d$. In the case of an essentially low-dimensional domain, we show an approximation rate

[TABLE]

for ReLU FNNs to approximate $f$ in the $\varepsilon$-neighborhood, d_{\delta}={\mathcal{O}}\big{(}d_{\mathcal{M}}\tfrac{\ln(d/\delta)}{\delta^{2}}\big{)} for any given $\delta\in(0,1)$.

Besides, we studied how to select the best ReLU FNN to approximate continuous function in parallel computing. In particular, ReLU FNNs with depth ${\mathcal{O}}(1)$ are the best choices if the number of parallel computing cores $p$ is sufficiently large. ReLU FNNs with width ${\mathcal{O}}(d)$ are best choices if $p={\mathcal{O}}(1)$. The width of best ReLU FNNs is between ${\mathcal{O}}(\sqrt{p})$ and ${\mathcal{O}}(p)$ if $p$ is moderate.

We would like to remark that our analysis was based on the fully connected feed-forward neural networks and the ReLU activation function. It would be very interesting to generalize our conclusions to neural networks with other types of architectures (e.g., convolutional neural networks) and activation functions (e.g., tanh and sigmoid functions). Besides, if identity maps are allowed in the construction of neural networks as in the residual networks [28], the size of FNNs in our construction can be further optimized. Finally, the proposed analysis could be generalized to other function spaces with explicit formulas to characterize the approximation error. These will be left as future work.

Acknowledgments

Z. Shen is supported by Tan Chin Tuan Centennial Professorship. H. Yang was partially supported by the US National Science Foundation under award DMS-1945029.