# Direct Synthesis of Iterative Algorithms With Bounds on Achievable Worst-Case Convergence Rate

## TL;DR

This paper introduces a novel method for directly synthesizing iterative algorithms with guaranteed worst-case convergence bounds, surpassing previous approaches by accommodating algorithms with arbitrary finite memory.

## Contribution

It proposes a robust control synthesis approach to directly design algorithms with performance guarantees, extending beyond fixed-structure algorithms.

## Key findings

- Stronger convergence bounds than previous methods.
- Applicable to algorithms with arbitrary finite memory.
- Demonstrates effectiveness through theoretical analysis.

## Abstract

Iterative first-order methods such as gradient descent and its variants are widely used for solving optimization and machine learning problems. There has been recent interest in analytic or numerically efficient methods for computing worst-case performance bounds for such algorithms, for example over the class of strongly convex loss functions. A popular approach is to assume the algorithm has a fixed size (fixed dimension, or memory) and that its structure is parameterized by one or two hyperparameters, for example a learning rate and a momentum parameter. Then, a Lyapunov function is sought to certify robust stability and subsequent optimization can be performed to find optimal hyperparameter tunings. In the present work, we instead fix the constraints that characterize the loss function and apply techniques from robust control synthesis to directly search over algorithms. This approach yields stronger results than those previously available, since the bounds produced hold over algorithms with an arbitrary, but finite, amount of memory rather than just holding for algorithms with a prescribed structure.