Approximation bounds for norm constrained deep neural networks

TL;DR

This paper establishes theoretical bounds on the approximation capabilities of norm-constrained neural networks with arbitrary activations, providing insights into their efficiency for smooth functions and validating results numerically.

Contribution

It introduces new upper and lower bounds on approximation errors for norm-constrained neural networks, extending understanding of their theoretical limits.

Findings

Upper bounds achieved via monomial approximation and Taylor expansion

Lower bounds derived from Rademacher complexity analysis

Numerical experiments confirm theoretical bounds

Abstract

This paper studies the approximation capacity of neural networks with an arbitrary activation function and with norm constraint on the weights. Upper and lower bounds on the approximation error of these networks are computed for smooth function classes. The upper bound is proven by first approximating high-degree monomials and then generalizing it to functions via a partition of unity and Taylor expansion. The lower bound is derived through the Rademacher complexity of neural networks. A probabilistic version of the upper bound is also provided by considering neural networks with randomly sampled weights and biases. Finally, it is shown that the assumption on the regularity of the activation function can be significantly weakened without worsening the approximation error, and the approximation upper bound is validated with numerical experiments.