From differential equation solvers to accelerated first-order methods for convex optimization

TL;DR

This paper introduces a novel dynamical system perspective on accelerated convex optimization methods, deriving new algorithms with proven accelerated convergence rates through differential equation analysis.

Contribution

It develops the Nesterov accelerated gradient flow from ODE stability principles, unifies existing methods, and creates new accelerated algorithms for composite convex optimization.

Findings

Proves exponential decay of a Lyapunov function for the new dynamical system.

Establishes convergence rates for discretized algorithms via a unified Lyapunov function.

Derives new accelerated algorithms with provable convergence for composite convex problems.

Abstract

Convergence analysis of accelerated first-order methods for convex optimization problems are presented from the point of view of ordinary differential equation solvers. A new dynamical system, called Nesterov accelerated gradient flow, has been derived from the connection between acceleration mechanism and $A$-stability of ODE solvers, and the exponential decay of a tailored Lyapunov function along with the solution trajectory is proved. Numerical discretizations are then considered and convergence rates are established via a unified discrete Lyapunov function. The proposed differential equation solver approach can not only cover existing accelerated methods, such as FISTA, G\"{u}ler's proximal algorithm and Nesterov's accelerated gradient method, but also produce new algorithms for composite convex optimization that possess accelerated convergence rates.