Qualitative neural network approximation over R and C: Elementary proofs for analytic and polynomial activation

TL;DR

This paper provides elementary proofs for approximation theorems of neural networks with analytic activation functions, showing their closure properties relate to polynomial spaces in both real and complex settings.

Contribution

It offers new elementary proofs for approximation theorems, extending results to networks with harmonic activation and residual architectures.

Findings

Neural networks with non-polynomial analytic activation functions can approximate polynomials.

Closure of neural network classes matches polynomial spaces, characterized by classical theorems.

Deep residual networks with polynomial activation can approximate any polynomial given sufficient width.

Abstract

In this article, we prove approximation theorems in classes of deep and shallow neural networks with analytic activation functions by elementary arguments. We prove for both real and complex networks with non-polynomial activation that the closure of the class of neural networks coincides with the closure of the space of polynomials. The closure can further be characterized by the Stone-Weierstrass theorem (in the real case) and Mergelyan's theorem (in the complex case). In the real case, we further prove approximation results for networks with higher-dimensional harmonic activation and orthogonally projected linear maps. We further show that fully connected and residual networks of large depth with polynomial activation functions can approximate any polynomial under certain width requirements. All proofs are entirely elementary.