Approximation capabilities of neural networks on unbounded domains

TL;DR

This paper investigates the approximation capabilities of shallow and deep neural networks with various activation functions on unbounded domains, establishing their universality and limitations in approximating integrable functions.

Contribution

It proves that shallow networks with common activations can approximate any L^p functions on bounded domains but not on unbounded planes, and shows deep ReLU networks' universal approximation in L^p.

Findings

Shallow networks can approximate L^p functions on bounded domains.

Shallow networks cannot approximate nonzero functions on the Euclidean plane.

Deep ReLU networks with depth 3 are universal in L^p(R^n).

Abstract

In this paper, we prove that a shallow neural network with a monotone sigmoid, ReLU, ELU, Softplus, or LeakyReLU activation function can arbitrarily well approximate any L^p(p>=2) integrable functions defined on R*[0,1]^n. We also prove that a shallow neural network with a sigmoid, ReLU, ELU, Softplus, or LeakyReLU activation function expresses no nonzero integrable function defined on the Euclidean plane. Together with a recent result that the deep ReLU network can arbitrarily well approximate any integrable function on Euclidean spaces, we provide a new perspective on the advantage of multiple hidden layers in the context of ReLU networks. Lastly, we prove that the ReLU network with depth 3 is a universal approximator in L^p(R^n).