# Analysis of Optimization Algorithms via Sum-of-Squares

## TL;DR

This paper introduces a sum-of-squares (SOS) based framework to analyze and certify convergence rates of first-order convex optimization algorithms, unifying existing methods and providing new bounds especially for noisy gradient descent.

## Contribution

It establishes a hierarchy of semidefinite programs for convergence analysis, connecting SOS proofs with the Performance Estimation Problem framework, and derives new bounds for noisy gradient descent.

## Key findings

- First level SOS hierarchy corresponds to the PEP framework.
- New convergence bounds for noisy gradient descent with inexact line search.
- SOS framework offers a systematic approach to certify improved convergence rates.

## Abstract

We introduce a new framework for unifying and systematizing the performance analysis of first-order black-box optimization algorithms for unconstrained convex minimization. The low-cost iteration complexity enjoyed by first-order algorithms renders them particularly relevant for applications in machine learning and large-scale data analysis. Relying on sum-of-squares (SOS) optimization, we introduce a hierarchy of semidefinite programs that give increasingly better convergence bounds for higher levels of the hierarchy. Alluding to the power of the SOS hierarchy, we show that the (dual of the) first level corresponds to the Performance Estimation Problem (PEP) introduced by Drori and Teboulle [Math. Program., 145(1):451--482, 2014], a powerful framework for determining convergence rates of first-order optimization algorithms. Consequently, many results obtained within the PEP framework can be reinterpreted as degree-1 SOS proofs, and thus, the SOS framework provides a promising new approach for certifying improved rates of convergence by means of higher-order SOS certificates. To determine analytical rate bounds, in this work we use the first level of the SOS hierarchy and derive new result{s} for noisy gradient descent with inexact line search methods (Armijo, Wolfe, and Goldstein).