From Averaging to Acceleration, There is Only a Step-size

TL;DR

This paper unifies accelerated gradient methods into second-order difference equations, analyzing their stability and convergence, and proposes an alternative algorithm suitable for noisy gradient scenarios.

Contribution

It reformulates key optimization algorithms as second-order difference equations and provides a detailed stability and convergence analysis, including noisy gradient cases.

Findings

Accelerated methods can be represented as second-order difference equations.

Stability of these systems is linked to convergence rate O(1/n^2).

An alternative algorithm with improved properties for noisy gradients is proposed.

Abstract

We show that accelerated gradient descent, averaged gradient descent and the heavy-ball method for non-strongly-convex problems may be reformulated as constant parameter second-order difference equation algorithms, where stability of the system is equivalent to convergence at rate O(1/n 2), where n is the number of iterations. We provide a detailed analysis of the eigenvalues of the corresponding linear dynamical system , showing various oscillatory and non-oscillatory behaviors, together with a sharp stability result with explicit constants. We also consider the situation where noisy gradients are available, where we extend our general convergence result, which suggests an alternative algorithm (i.e., with different step sizes) that exhibits the good aspects of both averaging and acceleration.