Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces

TL;DR

This paper establishes bounds on the approximation error of ReLU neural networks for low-regularity functions, linking error to network size and function norm, and provides a constructive proof via Fourier features residual networks.

Contribution

It introduces a novel approximation error bound for ReLU networks on low-regularity functions, derived from Fourier features residual networks, with a detailed complexity analysis.

Findings

Approximation error is proportional to the function's uniform norm.

Error inversely related to network width and depth.

Constructive proof via Fourier features residual networks.

Abstract

In this work, we consider the approximation of a large class of bounded functions, with minimal regularity assumptions, by ReLU neural networks. We show that the approximation error can be bounded from above by a quantity proportional to the uniform norm of the target function and inversely proportional to the product of network width and depth. We inherit this approximation error bound from Fourier features residual networks, a type of neural network that uses complex exponential activation functions. Our proof is constructive and proceeds by conducting a careful complexity analysis associated with the approximation of a Fourier features residual network by a ReLU network.