Efficient approximation of high-dimensional functions with neural networks

TL;DR

This paper introduces a framework using catalog networks to demonstrate that neural networks can efficiently approximate high-dimensional functions, overcoming the curse of dimensionality, with precise parameter estimates.

Contribution

The paper presents a novel catalog network framework allowing variable activation functions, enabling efficient high-dimensional function approximation with ReLU networks.

Findings

Catalog networks can be approximated by ReLU networks with explicit parameter bounds.

Certain classes of high-dimensional functions can be approximated without the curse of dimensionality.

The approach provides precise estimates on the number of parameters needed for a given accuracy.

Abstract

In this paper, we develop a framework for showing that neural networks can overcome the curse of dimensionality in different high-dimensional approximation problems. Our approach is based on the notion of a catalog network, which is a generalization of a standard neural network in which the nonlinear activation functions can vary from layer to layer as long as they are chosen from a predefined catalog of functions. As such, catalog networks constitute a rich family of continuous functions. We show that under appropriate conditions on the catalog, catalog networks can efficiently be approximated with rectified linear unit-type networks and provide precise estimates on the number of parameters needed for a given approximation accuracy. As special cases of the general results, we obtain different classes of functions that can be approximated with ReLU networks without the curse of dimensionality.