Nonclosedness of Sets of Neural Networks in Sobolev Spaces

TL;DR

This paper investigates the nonclosedness of neural network sets in Sobolev spaces, revealing limitations in their approximation capabilities and providing insights into parameter growth and training performance.

Contribution

It demonstrates that neural network sets are not closed in Sobolev spaces, characterizes parameter growth rates, and shows neural networks can approximate non-network functions with increasing parameters.

Findings

Neural network sets are not closed in Sobolev spaces $W^{m-1,p}$ and $W^{m,p}$.

Sequences of neural networks can approximate functions outside the network realizations.

Networks can closely approximate non-network target functions with increasing parameters.

Abstract

We examine the closedness of sets of realized neural networks of a fixed architecture in Sobolev spaces. For an exactly $m$-times differentiable activation function $\rho$, we construct a sequence of neural networks $(\Phi_n)_{n \in \mathbb{N}}$ whose realizations converge in order-$(m-1)$ Sobolev norm to a function that cannot be realized exactly by a neural network. Thus, sets of realized neural networks are not closed in order-$(m-1)$ Sobolev spaces $W^{m-1,p}$ for $p \in [1,\infty]$. We further show that these sets are not closed in $W^{m,p}$ under slightly stronger conditions on the $m$-th derivative of $\rho$. For a real analytic activation function, we show that sets of realized neural networks are not closed in $W^{k,p}$ for any $k \in \mathbb{N}$. The nonclosedness allows for approximation of non-network target functions with unbounded parameter growth. We partially characterize the rate of parameter growth for most activation functions by showing that a specific sequence of realized neural networks can approximate the activation function's derivative with weights increasing inversely proportional to the $L^p$ approximation error. Finally, we present experimental results showing that networks are capable of closely approximating non-network target functions with increasing parameters via training.