A note on sample complexity of learning binary output neural networks under fixed input distributions

TL;DR

This paper demonstrates that the sample complexity for learning certain neural networks can grow arbitrarily fast depending on the input distribution, highlighting limitations in learnability under fixed distributions.

Contribution

It establishes that the sample complexity of Sontag's sigmoidal neural network can grow arbitrarily fast for specific input distributions, and shows it is not Glivenko-Cantelli under non-atomic distributions.

Findings

Sample complexity can grow superexponentially with input distribution.

Sontag's neural network is not Glivenko-Cantelli under non-atomic distributions.

Asymptotically tight bounds on sample complexity growth.

Abstract

We show that the learning sample complexity of a sigmoidal neural network constructed by Sontag (1992) required to achieve a given misclassification error under a fixed purely atomic distribution can grow arbitrarily fast: for any prescribed rate of growth there is an input distribution having this rate as the sample complexity, and the bound is asymptotically tight. The rate can be superexponential, a non-recursive function, etc. We further observe that Sontag's ANN is not Glivenko-Cantelli under any input distribution having a non-atomic part.