On the Saturation Phenomenon of Stochastic Gradient Descent for Linear Inverse Problems

TL;DR

This paper refines the understanding of stochastic gradient descent (SGD) for linear inverse problems, showing that saturation in convergence rates can be avoided with a small enough initial stepsize, supported by theoretical analysis and experiments.

Contribution

It provides a refined convergence rate analysis of SGD, demonstrating that saturation does not occur if the initial stepsize is sufficiently small.

Findings

Saturation phenomenon can be avoided with small initial stepsize.

Refined convergence rates are established for SGD.

Numerical experiments support the theoretical results.

Abstract

Stochastic gradient descent (SGD) is a promising method for solving large-scale inverse problems, due to its excellent scalability with respect to data size. The current mathematical theory in the lens of regularization theory predicts that SGD with a polynomially decaying stepsize schedule may suffer from an undesirable saturation phenomenon, i.e., the convergence rate does not further improve with the solution regularity index when it is beyond a certain range. In this work, we present a refined convergence rate analysis of SGD, and prove that saturation actually does not occur if the initial stepsize of the schedule is sufficiently small. Several numerical experiments are provided to complement the analysis.