Nesterov's method with decreasing learning rate leads to accelerated stochastic gradient descent

TL;DR

This paper derives new stochastic gradient descent algorithms from a coupled ODE system, demonstrating accelerated convergence rates with decreasing learning rates in both convex and strongly convex settings.

Contribution

It introduces novel SGD algorithms based on ODE discretization with decreasing learning rates, achieving optimal convergence rates and improved constants.

Findings

Derived SGD algorithms with accelerated convergence.

Proved convergence at optimal rates for last iterate.

Achieved better rate constants than previous methods.

Abstract

We present a coupled system of ODEs which, when discretized with a constant time step/learning rate, recovers Nesterov's accelerated gradient descent algorithm. The same ODEs, when discretized with a decreasing learning rate, leads to novel stochastic gradient descent (SGD) algorithms, one in the convex and a second in the strongly convex case. In the strongly convex case, we obtain an algorithm superficially similar to momentum SGD, but with additional terms. In the convex case, we obtain an algorithm with a novel order $k^{3/4}$ learning rate. We prove, extending the Lyapunov function approach from the full gradient case to the stochastic case, that the algorithms converge at the optimal rate for the last iterate of SGD, with rate constants which are better than previously available.