Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints

TL;DR

This paper introduces a novel framework using Integral Quadratic Constraints from control theory to analyze and design iterative optimization algorithms, providing new convergence bounds and a methodology for creating algorithms with specific performance goals.

Contribution

It adapts IQC theory for optimization, deriving new inequalities and bounds, and proposes a new approach for algorithm design using semidefinite programming.

Findings

Derived numerical upper bounds on convergence rates for various optimization methods.

Provided a new methodology for designing algorithms with desired performance.

Extended IQC theory to the context of optimization algorithms.

Abstract

This manuscript develops a new framework to analyze and design iterative optimization algorithms built on the notion of Integral Quadratic Constraints (IQC) from robust control theory. IQCs provide sufficient conditions for the stability of complicated interconnected systems, and these conditions can be checked by semidefinite programming. We discuss how to adapt IQC theory to study optimization algorithms, proving new inequalities about convex functions and providing a version of IQC theory adapted for use by optimization researchers. Using these inequalities, we derive numerical upper bounds on convergence rates for the gradient method, the heavy-ball method, Nesterov's accelerated method, and related variants by solving small, simple semidefinite programming problems. We also briefly show how these techniques can be used to search for optimization algorithms with desired performance characteristics, establishing a new methodology for algorithm design.