Function and derivative approximation by shallow neural networks

TL;DR

This paper develops a Tikhonov regularization approach for shallow neural networks to improve function and derivative approximation, analyzing the role of various network norms and their impact on approximation accuracy.

Contribution

It introduces a novel regularization scheme incorporating multiple network norms and provides rigorous error analysis and insights into the influence of network dimensionality.

Findings

Established connections between network norms and approximation capabilities

Provided error bounds for the regularization scheme

Analyzed the impact of dimensionality on approximation performance

Abstract

We investigate a Tikhonov regularization scheme specifically tailored for shallow neural networks within the context of solving a classic inverse problem: approximating an unknown function and its derivatives within a unit cubic domain based on noisy measurements. The proposed Tikhonov regularization scheme incorporates a penalty term that takes three distinct yet intricately related network (semi)norms: the extended Barron norm, the variation norm, and the Radon-BV seminorm. These choices of the penalty term are contingent upon the specific architecture of the neural network being utilized. We establish the connection between various network norms and particularly trace the dependence of the dimensionality index, aiming to deepen our understanding of how these norms interplay with each other. We revisit the universality of function approximation through various norms, establish rigorous error-bound analysis for the Tikhonov regularization scheme, and explicitly elucidate the dependency of the dimensionality index, providing a clearer understanding of how the dimensionality affects the approximation performance and how one designs a neural network with diverse approximating tasks.