Unified Optimal Analysis of the (Stochastic) Gradient Method

TL;DR

This paper provides a unified analysis of stochastic and deterministic gradient methods, establishing convergence rates under mild smoothness assumptions and matching the best known iteration complexities.

Contribution

It offers a simple proof for convergence of SGD under milder smoothness conditions and unifies the analysis for both stochastic and deterministic gradient methods.

Findings

Convergence rate for SGD with $ ilde{O}(LR^2 e^{-rac{}{4L}T} + rac{\sigma^2}{ T})$.

Recovery of exponential convergence in the interpolation setting where $\sigma^2=0$.

Match with best known iteration complexity bounds for GD and SGD.

Abstract

In this note we give a simple proof for the convergence of stochastic gradient (SGD) methods on $\mu$-convex functions under a (milder than standard) $L$-smoothness assumption. We show that for carefully chosen stepsizes SGD converges after $T$ iterations as $O\left( LR^2 \exp \bigl[-\frac{\mu}{4L}T\bigr] + \frac{\sigma^2}{\mu T} \right)$ where $\sigma^2$ measures the variance in the stochastic noise. For deterministic gradient descent (GD) and SGD in the interpolation setting we have $\sigma^2 =0$ and we recover the exponential convergence rate. The bound matches with the best known iteration complexity of GD and SGD, up to constants.