Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime

TL;DR

This paper proves that stochastic gradient descent (SGD) with large stepsizes converges rapidly in the last iterate for smooth convex functions in the interpolation regime, with near-optimal rates that improve upon previous results.

Contribution

It establishes new convergence guarantees for the last iterate of SGD with large stepsizes in the interpolation regime, extending prior work beyond least squares regression.

Findings

Expected excess risk of SGD last iterate is near optimal, combining $1/T$ and $rac{ ext{noise}}{ ext{sqrt}(T)}$ rates.

For zero noise, SGD last iterate achieves an $O(1/ oot T)$ convergence rate.

Improves upon previous rates in realizable linear regression, especially with large stepsizes.

Abstract

We study population convergence guarantees of stochastic gradient descent (SGD) for smooth convex objectives in the interpolation regime, where the noise at optimum is zero or near zero. The behavior of the last iterate of SGD in this setting -- particularly with large (constant) stepsizes -- has received growing attention in recent years due to implications for the training of over-parameterized models, as well as to analyzing forgetting in continual learning and to understanding the convergence of the randomized Kaczmarz method for solving linear systems. We establish that after $T$ steps of SGD on $\beta$-smooth convex loss functions with stepsize $0 < \eta < 2/\beta$, the last iterate exhibits expected excess risk $\widetilde{O}(\frac{1}{\eta (2-\beta \eta) T^{1-\beta\eta/2}} + \frac{\eta}{(2-\beta\eta)^2} T^{\beta\eta/2} \sigma_\star^2)$, where $\sigma_\star^2$ denotes the variance of the stochastic gradients at the optimum. In particular, for a well-tuned stepsize we obtain a near optimal $\widetilde{O}(1/T + \sigma_\star/\sqrt{T})$ rate for the last iterate, extending the results of Varre et al. (2021) beyond least squares regression; and when $\sigma_\star=0$ we obtain a rate of $\smash{O(1/\sqrt T)}$ with $\eta=1/\beta$, improving upon the best-known $\smash{O(T^{-1/4})}$ rate recently established by Evron et al. (2025) in the special case of realizable linear regression.