Sharp uniform approximation for spectral Barron functions by deep neural networks

TL;DR

This paper establishes sharp uniform approximation rates for spectral Barron functions using shallow and deep neural networks, reducing smoothness requirements and demonstrating the benefits of depth for small smoothness functions.

Contribution

The work provides new approximation rates for spectral Barron functions with neural networks, lowering smoothness assumptions and highlighting depth's role in approximation quality.

Findings

Shallow networks achieve $O(N^{-1/2})$ approximation for $B^{1/2}$ functions.

Deeper networks improve approximation rates for functions with small smoothness.

Rates are dimension-free and nearly optimal, with tight lower bounds.

Abstract

This work explores the neural network approximation capabilities for functions within the spectral Barron space $\mathscr{B}^s$, where $s$ is the smoothness index. We demonstrate that for functions in $\mathscr{B}^{1/2}$, a shallow neural network (a single hidden layer) with $N$ units can achieve an $L^p$-approximation rate of $\mathcal{O}(N^{-1/2})$. This rate also applies to uniform approximation, differing by at most a logarithmic factor. Our results significantly reduce the smoothness requirement compared to existing theory, which necessitate functions to belong to $\mathscr{B}^1$ in order to attain the same rate. Furthermore, we show that increasing the network's depth can notably improve the approximation order for functions with small smoothness. Specifically, for networks with $L$ hidden layers, functions in $\mathscr{B}^s$ with $0 < sL \le 1/2$ can achieve an approximation rate of $\mathcal{O}(N^{-sL})$. The rates and prefactors in our estimates are dimension-free. We also confirm the sharpness of our findings, with the lower bound closely aligning with the upper, with a discrepancy of at most one logarithmic factor.