Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networks

TL;DR

This paper extends the universal approximation theorem to operator neural networks with bounded width and arbitrary depth, demonstrating their ability to approximate continuous nonlinear operators and highlighting depth's advantages.

Contribution

It proves that bounded-width, arbitrarily deep operator NNs can universally approximate continuous nonlinear operators, including for various activation functions.

Findings

Operator NNs of width five can approximate any continuous nonlinear operator.

Depth provides theoretical advantages in approximation capacity.

ReLU NNs of certain depths cannot be approximated by shallower networks unless width is exponential.

Abstract

The standard Universal Approximation Theorem for operator neural networks (NNs) holds for arbitrary width and bounded depth. Here, we prove that operator NNs of bounded width and arbitrary depth are universal approximators for continuous nonlinear operators. In our main result, we prove that for non-polynomial activation functions that are continuously differentiable at a point with a nonzero derivative, one can construct an operator NN of width five, whose inputs are real numbers with finite decimal representations, that is arbitrarily close to any given continuous nonlinear operator. We derive an analogous result for non-affine polynomial activation functions. We also show that depth has theoretical advantages by constructing operator ReLU NNs of depth $2k^3+8$ and constant width that cannot be well-approximated by any operator ReLU NN of depth $k$, unless its width is exponential in $k$.