Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regression

TL;DR

This paper demonstrates that over-parameterized neural networks trained with gradient descent and early stopping achieve optimal nonparametric regression rates similar to kernel methods, without restrictive distributional assumptions on covariates.

Contribution

It establishes that gradient descent training of over-parameterized neural networks attains minimax optimal rates for nonparametric regression under broad conditions, bridging neural networks and kernel methods.

Findings

Achieves $oxed{ ext{O}( ext{} ext{ extepsilon}_n^2)}$ regression risk rate.

No distributional assumptions on covariates needed, only boundedness.

Provides insights on stopping time, network width, and learning rate for training.

Abstract

We study nonparametric regression by an over-parameterized two-layer neural network trained by gradient descent (GD) in this paper. We show that, if the neural network is trained by GD with early stopping, then the trained network renders a sharp rate of the nonparametric regression risk of $\mathcal{O}(\epsilon_n^2)$, which is the same rate as that for the classical kernel regression trained by GD with early stopping, where $\epsilon_n$ is the critical population rate of the Neural Tangent Kernel (NTK) associated with the network and $n$ is the size of the training data. It is remarked that our result does not require distributional assumptions about the covariate as long as the covariate is bounded, in a strong contrast with many existing results which rely on specific distributions of the covariates such as the spherical uniform data distribution or distributions satisfying certain restrictive conditions. The rate $\mathcal{O}(\epsilon_n^2)$ is known to be minimax optimal for specific cases, such as the case that the NTK has a polynomial eigenvalue decay rate which happens under certain distributional assumptions on the covariates. Our result formally fills the gap between training a classical kernel regression model and training an over-parameterized but finite-width neural network by GD for nonparametric regression without distributional assumptions on the bounded covariate. We also provide confirmative answers to certain open questions or address particular concerns in the literature of training over-parameterized neural networks by GD with early stopping for nonparametric regression, including the characterization of the stopping time, the lower bound for the network width, and the constant learning rate used in GD.