The connections between Lyapunov functions for some optimization algorithms and differential equations

TL;DR

This paper explores the relationship between Lyapunov functions for certain optimization algorithms and differential equations, providing new analytical tools to characterize convergence rates and stability.

Contribution

It introduces a novel Lyapunov function framework for a family of Nesterov methods using LMI, linking discrete algorithms with continuous differential equations.

Findings

Derived a Lyapunov function for Nesterov methods using LMI

Characterized convergence rates of accelerated algorithms

Showed most discretizations lack similar Lyapunov functions

Abstract

In this manuscript, we study the properties of a family of second-order differential equations with damping, its discretizations and their connections with accelerated optimization algorithms for $m$-strongly convex and $L$-smooth functions. In particular, using the Linear Matrix Inequality LMI framework developed by \emph{Fazlyab et. al. $(2018)$}, we derive analytically a (discrete) Lyapunov function for a two-parameter family of Nesterov optimization methods, which allows for the complete characterization of their convergence rate. In the appropriate limit, this family of methods may be seen as a discretization of a family of second-order ordinary differential equations for which we construct(continuous) Lyapunov functions by means of the LMI framework. The continuous Lyapunov functions may alternatively, be obtained by studying the limiting behaviour of their discrete counterparts. Finally, we show that the majority of typical discretizations of the family of ODEs, such as the Heavy ball method, do not possess Lyapunov functions with properties similar to those of the Lyapunov function constructed here for the Nesterov method.