A Survey on Universal Approximation Theorems

TL;DR

This survey comprehensively reviews universal approximation theorems for neural networks, covering foundational mathematical results and their implications for network design and capabilities.

Contribution

It provides a systematic overview of UATs, integrating classical approximation theorems with neural network theory, highlighting both theoretical and numerical perspectives.

Findings

Summarizes key classical approximation theorems relevant to neural networks.

Analyzes the implications of UATs for network width and depth.

Highlights the theoretical foundations underpinning neural network approximation capabilities.

Abstract

This paper discusses various theorems on the approximation capabilities of neural networks (NNs), which are known as universal approximation theorems (UATs). The paper gives a systematic overview of UATs starting from the preliminary results on function approximation, such as Taylor's theorem, Fourier's theorem, Weierstrass approximation theorem, Kolmogorov - Arnold representation theorem, etc. Theoretical and numerical aspects of UATs are covered from both arbitrary width and depth.