Implicit vs. explicit regularization for high-dimensional gradient descent

TL;DR

This paper compares implicit and explicit regularization in high-dimensional gradient descent, showing explicit regularization's near-optimal statistical efficiency and proposing a data-driven method for selecting regularization parameters.

Contribution

It introduces a new methodology for high-dimensional linear prediction that demonstrates the advantages of explicit regularization over implicit early stopping, with a computationally efficient parameter choice.

Findings

Explicit regularization approaches the benchmark error with optimal tuning.

Implicit regularization via early stopping is less statistically efficient.

Proposes a data-driven method for regularization parameter selection.

Abstract

In this paper we investigate the generalization error of gradient descent (GD) applied to an $\ell_2$-regularized OLS objective function in the linear model. Based on our analysis we develop new methodology for computationally tractable and statistically efficient linear prediction in a high-dimensional and massive data scenario (large-$n$, large-$p$). Our results are based on the surprising observation that the generalization error of optimally tuned regularized gradient descent approaches that of an optimal benchmark procedure $monotonically$ in the iteration number $t$. On the other hand standard GD for OLS (without explicit regularization) can achieve the benchmark only in degenerate cases. This shows that (optimal) explicit regularization can be nearly statistically efficient (for large $t$) whereas implicit regularization by (optimal) early stopping can not. To complete our methodology, we provide a fully data driven and computationally tractable choice of the $\ell_2$ regularization parameter $\lambda$ that is computationally cheaper than cross-validation. On this way, we follow and extend ideas of Dicker (2014) to the non-gaussian case, which requires new results on high-dimensional sample covariance matrices that might be of independent interest.