High-dimensional approximation spaces of artificial neural networks and applications to partial differential equations

TL;DR

This paper introduces a new framework called approximation spaces to analyze how neural networks can efficiently approximate high-dimensional functions, demonstrating their ability to overcome the curse of dimensionality in solving certain PDEs.

Contribution

The paper develops the concept of approximation spaces for neural networks, providing tools to analyze their capacity to approximate high-dimensional functions without curse of dimensionality, and applies this to PDEs.

Findings

Neural networks can approximate functions in high dimensions with polynomial complexity.

Approximation spaces are stable under various mathematical operations.

Neural networks can overcome the curse of dimensionality in certain PDE approximations.

Abstract

In this paper we develop a new machinery to study the capacity of artificial neural networks (ANNs) to approximate high-dimensional functions without suffering from the curse of dimensionality. Specifically, we introduce a concept which we refer to as approximation spaces of artificial neural networks and we present several tools to handle those spaces. Roughly speaking, approximation spaces consist of sequences of functions which can, in a suitable way, be approximated by ANNs without curse of dimensionality in the sense that the number of required ANN parameters to approximate a function of the sequence with an accuracy $\varepsilon > 0$ grows at most polynomially both in the reciprocal $1/\varepsilon$ of the required accuracy and in the dimension $d \in \mathbb{N} = \{1, 2, 3, \ldots \}$ of the function. We show that these approximation spaces are closed under various operations including linear combinations, formations of limits, and infinite compositions. To illustrate the utility of the machinery proposed in this paper, we employ the developed theory to prove that ANNs have the capacity to overcome the curse of dimensionality in the numerical approximation of certain first order transport partial differential equations (PDEs). We even prove that approximation spaces are closed under flows of first order transport PDEs.