Sobolev-type embeddings for neural network approximation spaces

TL;DR

This paper establishes Sobolev-type embedding theorems for neural network approximation spaces, revealing how approximation rate, integrability, and smoothness relate, and implications for function learning from neural network models.

Contribution

It provides new embedding theorems between neural network approximation spaces and classical function spaces, extending understanding of their structure and learning implications.

Findings

Embedding theorems between neural network approximation spaces for different p-values

Sharp embeddings into Hölder spaces showing trade-offs between smoothness and integrability

Optimal learning algorithms can be simple piecewise constant interpolations for well-approximable functions

Abstract

We consider neural network approximation spaces that classify functions according to the rate at which they can be approximated (with error measured in $L^p$) by ReLU neural networks with an increasing number of coefficients, subject to bounds on the magnitude of the coefficients and the number of hidden layers. We prove embedding theorems between these spaces for different values of $p$. Furthermore, we derive sharp embeddings of these approximation spaces into H\"older spaces. We find that, analogous to the case of classical function spaces (such as Sobolev spaces, or Besov spaces) it is possible to trade "smoothness" (i.e., approximation rate) for increased integrability. Combined with our earlier results in [arXiv:2104.02746], our embedding theorems imply a somewhat surprising fact related to "learning" functions from a given neural network space based on point samples: if accuracy is measured with respect to the uniform norm, then an optimal "learning" algorithm for reconstructing functions that are well approximable by ReLU neural networks is simply given by piecewise constant interpolation on a tensor product grid.