A Tutorial on a Lyapunov-Based Approach to the Analysis of Iterative Optimization Algorithms

TL;DR

This paper presents a Lyapunov-based method for analyzing the performance of iterative gradient-based optimization algorithms, offering an alternative to IQC-based analysis and enabling the construction of Lyapunov functions.

Contribution

It introduces a Lyapunov-based approach for analyzing optimization algorithms, providing a new perspective and tools for performance assessment.

Findings

The Lyapunov approach recovers known performance bounds.

It allows explicit construction of Lyapunov functions.

Provides an alternative to IQC-based analysis.

Abstract

Iterative gradient-based optimization algorithms are widely used to solve difficult or large-scale optimization problems. There are many algorithms to choose from, such as gradient descent and its accelerated variants such as Polyak's Heavy Ball method or Nesterov's Fast Gradient method. It has long been observed that iterative algorithms can be viewed as dynamical systems, and more recently, as robust controllers. Here, the "uncertainty" in the dynamics is the gradient of the function being optimized. Therefore, worst-case or average-case performance can be analyzed using tools from robust control theory, such as integral quadratic constraints (IQCs). In this tutorial paper, we show how such an analysis can be carried out using an alternative Lyapunov-based approach. This approach recovers the same performance bounds as with IQCs, but with the added benefit of constructing a Lyapunov function.