Neural Network with Unbounded Activation Functions is Universal Approximator

TL;DR

This paper demonstrates that neural networks with unbounded activation functions like ReLU are universal approximators, and introduces a new perspective using ridgelet transforms to analyze their approximation capabilities.

Contribution

It provides a theoretical proof of the universal approximation property for unbounded activation functions using ridgelet analysis and offers a novel method for network construction without backpropagation.

Findings

ReLU networks satisfy the universal approximation property.

Ridgelet transform analysis reveals what networks learn after training.

Discretization of the ridgelet transform can construct trained networks without backpropagation.

Abstract

This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network learns after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by simply discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some non-admissible cases result in low-pass filtering.