How many samples are needed to train a deep neural network?

TL;DR

This paper investigates the data requirements for training ReLU neural networks, revealing that their generalization error decreases at a rate of 1/√n, indicating they need large datasets for effective learning.

Contribution

It provides theoretical and empirical evidence that ReLU neural networks' generalization error scales with 1/√n, challenging the assumption of parametric rate convergence.

Findings

Generalization error scales as 1/√n

Neural networks require large datasets for good performance

Empirical results support theoretical predictions

Abstract

Neural networks have become standard tools in many areas, yet many important statistical questions remain open. This paper studies the question of how much data are needed to train a ReLU feed-forward neural network. Our theoretical and empirical results suggest that the generalization error of ReLU feed-forward neural networks scales at the rate $1/\sqrt{n}$ in the sample size $n$ rather than the usual "parametric rate" $1/n$. Thus, broadly speaking, our results underpin the common belief that neural networks need "many" training samples.