Convergence Rate for the Last Iterate of Stochastic Gradient Descent Schemes

TL;DR

This paper analyzes the convergence rates of the last iterate of stochastic gradient descent (SGD) and stochastic heavy ball (SHB) methods for convex and non-convex functions with Hölder continuous gradients, using discrete Gronwall's inequality.

Contribution

It provides new convergence rate results for SGD and SHB without relying on Robbins-Siegmund theorem, including probabilistic bounds for convex functions with constant momentum.

Findings

SGD and SHB achieve specific convergence rates for non-convex objectives.

SHB with constant momentum attains a logarithmic convergence rate in probability for convex functions.

The paper recovers known results and extends them to broader settings with Hölder continuous gradients.

Abstract

We study the convergence rate for the last iterate of stochastic gradient descent (SGD) and stochastic heavy ball (SHB) in the parametric setting when the objective function $F$ is globally convex or non-convex whose gradient is $\gamma$-H\"{o}lder. Using only discrete Gronwall's inequality without Robbins-Siegmund theorem, we recover results for both SGD and SHB: $\min_{s\leq t} \|\nabla F(w_s)\|^2 = o(t^{p-1})$ for non-convex objectives and $F(w_{\tau \wedge t}) - F_* = o(t^{2\gamma/(1+\gamma) \cdot \max(p-1,-2p+1)-\epsilon})$ for $\beta \in (0, 1)$, $\tau := \inf \{ t > 0 : F(w_t) = F_*\}$, and $\min_{s \leq t} F(w_s) - F_* = o(t^{p-1})$ for convex objectives $F$ whose minimum is $F_*$. In addition, we proved that SHB with constant momentum parameter $\beta \in (0, 1)$ attains a convergence rate of $F(w_t) - F_* = O(t^{\max(p-1,-2p+1)} \log^2 \frac{t}{\delta})$ with probability at least $1-\delta$ when $F$ is convex and $\gamma = 1$ and step size $\alpha_t = \Theta(t^{-p})$ with $p \in (\frac{1}{2}, 1)$.