Neural Network Approximation

TL;DR

This paper surveys the approximation capabilities and stability of neural networks, especially ReLU-based ones, comparing them with traditional methods and analyzing their space filling properties and implications for learning efficiency.

Contribution

It provides a comprehensive analysis of neural network approximation properties, highlighting their unique space filling behavior and stability challenges compared to classical approximation techniques.

Findings

ReLU neural networks produce piecewise linear functions with complex partitions.

The output functions form a parameterized nonlinear manifold with space filling properties.

These properties enhance approximation ability but pose stability and numerical challenges.

Abstract

Neural Networks (NNs) are the method of choice for building learning algorithms. Their popularity stems from their empirical success on several challenging learning problems. However, most scholars agree that a convincing theoretical explanation for this success is still lacking. This article surveys the known approximation properties of the outputs of NNs with the aim of uncovering the properties that are not present in the more traditional methods of approximation used in numerical analysis. Comparisons are made with traditional approximation methods from the viewpoint of rate distortion. Another major component in the analysis of numerical approximation is the computational time needed to construct the approximation and this in turn is intimately connected with the stability of the approximation algorithm. So the stability of numerical approximation using NNs is a large part of the analysis put forward. The survey, for the most part, is concerned with NNs using the popular ReLU activation function. In this case, the outputs of the NNs are piecewise linear functions on rather complicated partitions of the domain of $f$ into cells that are convex polytopes. When the architecture of the NN is fixed and the parameters are allowed to vary, the set of output functions of the NN is a parameterized nonlinear manifold. It is shown that this manifold has certain space filling properties leading to an increased ability to approximate (better rate distortion) but at the expense of numerical stability. The space filling creates a challenge to the numerical method in finding best or good parameter choices when trying to approximate.