Rates of Approximation by ReLU Shallow Neural Networks

TL;DR

This paper establishes approximation rates for shallow ReLU neural networks, showing they can nearly match optimal rates for functions in Hölder spaces, especially in high dimensions.

Contribution

It provides the first known approximation rates for shallow ReLU networks approximating Hölder functions, bridging a gap in understanding their efficiency.

Findings

ReLU shallow networks with m neurons approximate Hölder functions at near-optimal rates.

The approximation rate depends logarithmically on the number of neurons and polynomially on the inverse of the approximation error.

Rates improve as the dimension increases, approaching the optimal rate in high-dimensional settings.

Abstract

Neural networks activated by the rectified linear unit (ReLU) play a central role in the recent development of deep learning. The topic of approximating functions from H\"older spaces by these networks is crucial for understanding the efficiency of the induced learning algorithms. Although the topic has been well investigated in the setting of deep neural networks with many layers of hidden neurons, it is still open for shallow networks having only one hidden layer. In this paper, we provide rates of uniform approximation by these networks. We show that ReLU shallow neural networks with $m$ hidden neurons can uniformly approximate functions from the H\"older space $W_\infty^r([-1, 1]^d)$ with rates $O((\log m)^{\frac{1}{2} +d}m^{-\frac{r}{d}\frac{d+2}{d+4}})$ when $r<d/2 +2$. Such rates are very close to the optimal one $O(m^{-\frac{r}{d}})$ in the sense that $\frac{d+2}{d+4}$ is close to $1$, when the dimension $d$ is large.