Lower bounds for artificial neural network approximations: A proof that shallow neural networks fail to overcome the curse of dimensionality

TL;DR

This paper proves that shallow neural networks cannot overcome the curse of dimensionality in high-dimensional function approximation, highlighting the importance of network depth for efficient approximation.

Contribution

It provides a concrete example of functions that require deep networks for efficient approximation, establishing a lower bound for shallow neural networks.

Findings

Deep ANNs can approximate certain high-dimensional functions without curse of dimensionality.

Shallow ANNs fail to approximate these functions efficiently, confirming the necessity of depth.

The work mathematically demonstrates the limitations of shallow neural networks in high-dimensional settings.

Abstract

Artificial neural networks (ANNs) have become a very powerful tool in the approximation of high-dimensional functions. Especially, deep ANNs, consisting of a large number of hidden layers, have been very successfully used in a series of practical relevant computational problems involving high-dimensional input data ranging from classification tasks in supervised learning to optimal decision problems in reinforcement learning. There are also a number of mathematical results in the scientific literature which study the approximation capacities of ANNs in the context of high-dimensional target functions. In particular, there are a series of mathematical results in the scientific literature which show that sufficiently deep ANNs have the capacity to overcome the curse of dimensionality in the approximation of certain target function classes in the sense that the number of parameters of the approximating ANNs grows at most polynomially in the dimension $d \in \mathbb{N}$ of the target functions under considerations. In the proofs of several of such high-dimensional approximation results it is crucial that the involved ANNs are sufficiently deep and consist a sufficiently large number of hidden layers which grows in the dimension of the considered target functions. It is the topic of this work to look a bit more detailed to the deepness of the involved ANNs in the approximation of high-dimensional target functions. In particular, the main result of this work proves that there exists a concretely specified sequence of functions which can be approximated without the curse of dimensionality by sufficiently deep ANNs but which cannot be approximated without the curse of dimensionality if the involved ANNs are shallow or not deep enough.