On Sharpness of Error Bounds for Single Hidden Layer Feedforward Neural Networks

TL;DR

This paper demonstrates the sharpness of error bounds in univariate approximation by single hidden layer neural networks with sigmoid and ReLU activations, using a novel extension of the uniform boundedness principle.

Contribution

It introduces a new non-linear extension of the uniform boundedness principle to establish the optimality of approximation error bounds for neural networks.

Findings

Error bounds are sharp and cannot be improved.

Approximation errors do not belong to the little-o class of bounds.

Results extend to various activation functions including inverse tangent.

Abstract

A new non-linear variant of a quantitative extension of the uniform boundedness principle is used to show sharpness of error bounds for univariate approximation by sums of sigmoid and ReLU functions. Single hidden layer feedforward neural networks with one input node perform such operations. Errors of best approximation can be expressed using moduli of smoothness of the function to be approximated (i.e., to be learned). In this context, the quantitative extension of the uniform boundedness principle indeed allows to construct counter examples that show approximation rates to be best possible. Approximation errors do not belong to the little-o class of given bounds. By choosing piecewise linear activation functions, the discussed problem becomes free knot spline approximation. Results of the present paper also hold for non-polynomial (and not piecewise defined) activation functions like inverse tangent. Based on Vapnik-Chervonenkis dimension, first results are shown for the logistic function.