A Lyapunov Analysis of Momentum Methods in Optimization

TL;DR

This paper establishes a connection between estimate sequences and Lyapunov functions, providing a unified framework for analyzing and developing momentum optimization algorithms, including new methods and insights into their continuous-time dynamics.

Contribution

It reveals an equivalence between estimate sequences and Lyapunov functions, simplifying analysis and enabling the creation of new momentum algorithms with a unified approach.

Findings

Unified Lyapunov-based analysis of momentum methods

Introduction of new momentum algorithms

Strengthened link between discrete algorithms and continuous dynamics

Abstract

Momentum methods play a significant role in optimization. Examples include Nesterov's accelerated gradient method and the conditional gradient algorithm. Several momentum methods are provably optimal under standard oracle models, and all use a technique called estimate sequences to analyze their convergence properties. The technique of estimate sequences has long been considered difficult to understand, leading many researchers to generate alternative, "more intuitive" methods and analyses. We show there is an equivalence between the technique of estimate sequences and a family of Lyapunov functions in both continuous and discrete time. This connection allows us to develop a simple and unified analysis of many existing momentum algorithms, introduce several new algorithms, and strengthen the connection between algorithms and continuous-time dynamical systems.