Optimal Approximation Complexity of High-Dimensional Functions with Neural Networks

TL;DR

This paper demonstrates that neural networks with ReLU and quadratic activations can optimally approximate high-dimensional functions, leveraging low-dimensional structures to overcome the curse of dimensionality.

Contribution

It introduces neural networks with combined ReLU and quadratic activations and proves their optimal approximation rates for various function classes, including Sobolev spaces.

Findings

Neural networks with ReLU and $x^2$ can approximate analytic and Sobolev functions arbitrarily well.

Optimal approximation rates are achieved across all nonlinear approximators, including standard ReLU networks.

Leveraging low local dimensionality helps overcome the curse of dimensionality in approximation tasks.

Abstract

We investigate properties of neural networks that use both ReLU and $x^2$ as activation functions and build upon previous results to show that both analytic functions and functions in Sobolev spaces can be approximated by such networks of constant depth to arbitrary accuracy, demonstrating optimal order approximation rates across all nonlinear approximators, including standard ReLU networks. We then show how to leverage low local dimensionality in some contexts to overcome the curse of dimensionality, obtaining approximation rates that are optimal for unknown lower-dimensional subspaces.