Analytic Regularity and Approximation Limits of Coefficient-Constrained Shallow Networks

TL;DR

This paper investigates the limitations of shallow neural networks with analytic activation functions under coefficient constraints, showing they cannot surpass polynomial approximation rates for non-analytic targets due to inherent regularity restrictions.

Contribution

It establishes a fundamental approximation barrier for coefficient-constrained shallow networks, extending classical polynomial approximation results to neural network settings.

Findings

Networks with analytic activations are limited by polynomial approximation rates.

Coefficient constraints induce strong regularity, preventing better approximation of non-analytic functions.

The results apply to smoother, Gevrey-regular activations, indicating a universal approximation barrier.

Abstract

We study approximation limits of single-hidden-layer neural networks with analytic activation functions under global coefficient constraints. Under uniform $\ell^1$ bounds, or more generally sub-exponential growth of the coefficients, we show that such networks generate model classes with strong quantitative regularity, leading to uniform analyticity of the realized functions. As a consequence, up to an exponentially small residual term, the error of best network approximation on generic target functions is bounded from below by the error of best polynomial approximation. In particular, networks with analytic activation functions with controlled coefficients cannot outperform classical polynomial approximation rates on non-analytic targets. The underlying rigidity phenomenon extends to smoother, non-analytic activations satisfying Gevrey-type regularity assumptions, yielding sub-exponential variants of the approximation barrier. The analysis is entirely deterministic and relies on a comparison argument combined with classical Bernstein-type estimates; extensions to higher dimensions are also discussed.