Tractability of approximation by general shallow networks

TL;DR

This paper improves bounds on how well shallow neural networks can approximate certain classes of functions, showing that approximation quality depends polynomially on the number of network units and not on the input space dimension.

Contribution

It provides sharper, dimension-independent approximation bounds for general shallow networks, extending previous results to broader function classes and network types.

Findings

Dimension-independent approximation bounds established

Constants depend polynomially on the dimensions

Applicable to various network architectures like ReLU, zonal functions, and radial basis functions

Abstract

In this paper, we present a sharper version of the results in the paper Dimension independent bounds for general shallow networks; Neural Networks, \textbf{123} (2020), 142-152. Let $\mathbb{X}$ and $\mathbb{Y}$ be compact metric spaces. We consider approximation of functions of the form $ x\mapsto\int_{\mathbb{Y}} G( x, y)d\tau( y)$, $ x\in\mathbb{X}$, by $G$-networks of the form $ x\mapsto \sum_{k=1}^n a_kG( x, y_k)$, $ y_1,\cdots, y_n\in\mathbb{Y}$, $a_1,\cdots, a_n\in\mathbb{R}$. Defining the dimensions of $\mathbb{X}$ and $\mathbb{Y}$ in terms of covering numbers, we obtain dimension independent bounds on the degree of approximation in terms of $n$, where also the constants involved are all dependent at most polynomially on the dimensions. Applications include approximation by power rectified linear unit networks, zonal function networks, certain radial basis function networks as well as the important problem of function extension to higher dimensional spaces.