Negative results for approximation using single layer and multilayer feedforward neural networks

TL;DR

This paper establishes fundamental limitations on the approximation capabilities of single-layer and multilayer feedforward neural networks for certain classes of functions, showing that some functions are inherently difficult to approximate regardless of network size or activation functions.

Contribution

It provides the first negative approximation results for both single-layer and multilayer neural networks with broad classes of activation functions, highlighting inherent limitations.

Findings

Existence of target functions that are hard to approximate with neural networks

Negative results apply to networks with arbitrary hidden layers and specific activation functions

Shows fundamental limitations in neural network approximation capabilities

Abstract

We prove a negative result for the approximation of functions defined on compact subsets of $\mathbb{R}^d$ (where $d \geq 2$) using feedforward neural networks with one hidden layer and arbitrary continuous activation function. In a nutshell, this result claims the existence of target functions that are as difficult to approximate using these neural networks as one may want. We also demonstrate an analogous result (for general $d \in \mathbb{N}$) for neural networks with an \emph{arbitrary} number of hidden layers, for activation functions that are either rational functions or continuous splines with finitely many pieces.