Density of Neural Network Classes on Compact Subsets of Topological Vector Spaces

TL;DR

This paper proves that neural network classes with certain activation functions are dense in continuous function spaces on compact subsets of topological vector spaces, extending classical approximation results.

Contribution

It establishes density of neural network classes in function spaces on topological vector spaces, generalizing known approximation theorems.

Findings

Neural network classes are dense in C(K) for compact K in topological vector spaces.

Density extends to L^p spaces with Radon probability measures.

Results hold for continuous squashing functions and dual spaces separating points.

Abstract

We prove density results for neural-network classes on compact sets \(K\subset X\), where \(X\) is a topological vector space whose continuous dual \(X^*\) separates points. Let \(\Psi:\mathbb R\to\mathbb R\) be a continuous squashing function. We show that the class \[ \Sigma_X(\Psi) = \left\{ \sum_{j=1}^{N}\omega_j\Psi(f_j(x)+b_j): N\in\mathbb N,\ \omega_j,b_j\in\mathbb R,\ f_j\in X^* \right\} \] is dense in \(C(K)\) with respect to the uniform norm. As a consequence, if \(\mu\) is a Radon probability measure supported on \(K\), then \(\Sigma_X(\Psi)\) is dense in \(L^p(K,\mu)\) for every \(1\le p<\infty\).