Implicit Regularization and Generalization in Overparameterized Neural Networks

TL;DR

This paper explores how optimization dynamics and implicit regularization enable overparameterized neural networks to generalize well, challenging classical theory predictions.

Contribution

It provides controlled experiments analyzing the effects of batch size, landscape geometry, NTK regime, double descent, and lottery tickets on generalization.

Findings

Smaller batch sizes lead to flatter minima and lower test error.

Sparse subnetworks can match full model performance with only 10% of parameters.

Flatter minima correlate with better generalization and lower Hessian eigenvalues.

Abstract

Classical statistical learning theory predicts that overparameterized models should exhibit severe overfitting, yet modern deep neural networks with far more parameters than training samples consistently generalize well. This contradiction has become a central theoretical question in machine learning. This study investigates the role of optimization dynamics and implicit regularization in enabling generalization in overparameterized neural networks through controlled experiments. We examine stochastic gradient descent (SGD) across batch sizes, the geometry of flat versus sharp minima via Hessian eigenvalue estimation and weight perturbation analysis, the Neural Tangent Kernel (NTK) regime through wide-network experiments, double descent across model scales, and the Lottery Ticket Hypothesis through iterative magnitude pruning. All experiments use PyTorch on CIFAR-10 and MNIST with multiple random seeds. Our findings demonstrate that generalization is strongly influenced by the interaction between network architecture, optimization algorithms, and loss landscape geometry. Smaller batch sizes consistently produced lower test error and flatter minima, with an 11.8x difference in top Hessian eigenvalue between small-batch and large-batch solutions corresponding to 1.61 percentage points higher test accuracy. Sparse subnetworks retaining only 10% of parameters achieved within 1.15 percentage points of full model performance when retrained from their original initialization. These results highlight the need for revised learning-theoretic frameworks capable of explaining generalization in high-dimensional model regimes.