On the Regularization Effect of Stochastic Gradient Descent applied to Least Squares

TL;DR

This paper analyzes how stochastic gradient descent (SGD) applied to least squares problems exhibits a regularization effect, especially when the residual aligns with large singular vectors, leading to smoothing of solutions.

Contribution

The paper provides an explicit inequality showing the regularization effect of SGD on least squares problems, with extensions to symmetric matrices and Sobolev spaces.

Findings

SGD induces a regularization effect depending on the residual's singular vector composition.

The inequality reveals a smoothing energy cascade from large to small singular values.

Extensions to symmetric matrices demonstrate higher-order Sobolev space regularization.

Abstract

We study the behavior of stochastic gradient descent applied to $\|Ax -b \|_2^2 \rightarrow \min$ for invertible $A \in \mathbb{R}^{n \times n}$. We show that there is an explicit constant $c_{A}$ depending (mildly) on $A$ such that $$ \mathbb{E} ~\left\| Ax_{k+1}-b\right\|^2_{2} \leq \left(1 + \frac{c_{A}}{\|A\|_F^2}\right) \left\|A x_k -b \right\|^2_{2} - \frac{2}{\|A\|_F^2} \left\|A^T A (x_k - x)\right\|^2_{2}.$$ This is a curious inequality: the last term has one more matrix applied to the residual $u_k - u$ than the remaining terms: if $x_k - x$ is mainly comprised of large singular vectors, stochastic gradient descent leads to a quick regularization. For symmetric matrices, this inequality has an extension to higher-order Sobolev spaces. This explains a (known) regularization phenomenon: an energy cascade from large singular values to small singular values smoothes.