How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent

TL;DR

This paper provides a detailed analysis of the number of neurons needed in shallow neural networks trained with gradient descent, establishing optimal convergence rates and conditions for generalization in the NTK regime.

Contribution

It refines existing bounds on neuron count for generalization, tracks weight proximity during training, and connects structural assumptions to training dynamics.

Findings

Fast convergence rates for early stopped GD in NTK regime

Precise neuron count requirements for generalization

Weights stay close to initialization during training

Abstract

We analyze the generalization properties of two-layer neural networks in the neural tangent kernel (NTK) regime, trained with gradient descent (GD). For early stopped GD we derive fast rates of convergence that are known to be minimax optimal in the framework of non-parametric regression in reproducing kernel Hilbert spaces. On our way, we precisely keep track of the number of hidden neurons required for generalization and improve over existing results. We further show that the weights during training remain in a vicinity around initialization, the radius being dependent on structural assumptions such as degree of smoothness of the regression function and eigenvalue decay of the integral operator associated to the NTK.