Approximation of Smoothness Classes by Deep Rectifier Networks

TL;DR

This paper demonstrates that deep rectifier neural networks can optimally approximate functions in Besov spaces across various smoothness levels and dimensions, highlighting their effectiveness in function approximation tasks.

Contribution

It establishes near-optimal approximation rates of deep ReLU and RePU networks for Besov space functions on arbitrary domains and dimensions.

Findings

Deep networks achieve optimal approximation rates on the critical embedding line.

Approximation extends to the entire range of smoothness classes above the critical line.

Results apply to functions in Besov spaces with arbitrary smoothness and integrability parameters.

Abstract

We consider approximation rates of sparsely connected deep rectified linear unit (ReLU) and rectified power unit (RePU) neural networks for functions in Besov spaces $B^\alpha_{q}(L^p)$ in arbitrary dimension $d$, on general domains. We show that \alert{deep rectifier} networks with a fixed activation function attain optimal or near to optimal approximation rates for functions in the Besov space $B^\alpha_{\tau}(L^\tau)$ on the critical embedding line $1/\tau=\alpha/d+1/p$ for \emph{arbitrary} smoothness order $\alpha>0$. Using interpolation theory, this implies that the entire range of smoothness classes at or above the critical line is (near to) optimally approximated by deep ReLU/RePU networks.