Dimension Independent Generalization Error by Stochastic Gradient Descent

TL;DR

This paper develops a theory showing that stochastic gradient descent solutions can generalize well in high-dimensional settings, especially when data and models have low effective dimension, explaining the success of overparameterized neural networks.

Contribution

It introduces a general framework for understanding the generalization error of SGD in high-dimensional models, highlighting conditions for low effective dimension and benign overfitting.

Findings

Generalization error can be independent of ambient dimension under certain conditions.

Low effective dimension naturally occurs in overparameterized models like neural networks.

The theory applies to both convex and non-convex models, including linear and neural network models.

Abstract

One classical canon of statistics is that large models are prone to overfitting, and model selection procedures are necessary for high dimensional data. However, many overparameterized models, such as neural networks, perform very well in practice, although they are often trained with simple online methods and regularization. The empirical success of overparameterized models, which is often known as benign overfitting, motivates us to have a new look at the statistical generalization theory for online optimization. In particular, we present a general theory on the generalization error of stochastic gradient descent (SGD) solutions for both convex and locally convex loss functions. We further discuss data and model conditions that lead to a ``low effective dimension". Under these conditions, we show that the generalization error either does not depend on the ambient dimension $p$ or depends on $p$ via a poly-logarithmic factor. We also demonstrate that in several widely used statistical models, the ``low effective dimension'' arises naturally in overparameterized settings. The studied statistical applications include both convex models such as linear regression and logistic regression and non-convex models such as $M$-estimator and two-layer neural networks.