On Regularization via Early Stopping for Least Squares Regression

TL;DR

This paper provides a detailed analysis of how early stopping acts as a form of regularization in linear regression, characterizing the parameter dynamics and demonstrating its benefits across various data spectra and learning rate schedules.

Contribution

It offers a theoretical characterization of early stopping as a regularization method in linear regression and derives an estimate for optimal stopping time.

Findings

Early stopping is equivalent to a generalized ridge regularization.

It benefits models trained on data with arbitrary spectral properties.

An accurate estimate for the optimal stopping time is provided.

Abstract

A fundamental problem in machine learning is understanding the effect of early stopping on the parameters obtained and the generalization capabilities of the model. Even for linear models, the effect is not fully understood for arbitrary learning rates and data. In this paper, we analyze the dynamics of discrete full batch gradient descent for linear regression. With minimal assumptions, we characterize the trajectory of the parameters and the expected excess risk. Using this characterization, we show that when training with a learning rate schedule $\eta_k$, and a finite time horizon $T$, the early stopped solution $\beta_T$ is equivalent to the minimum norm solution for a generalized ridge regularized problem. We also prove that early stopping is beneficial for generic data with arbitrary spectrum and for a wide variety of learning rate schedules. We provide an estimate for the optimal stopping time and empirically demonstrate the accuracy of our estimate.