Design of First-Order Optimization Algorithms via Sum-of-Squares Programming

TL;DR

This paper introduces a sum-of-squares programming framework to systematically design and optimize first-order algorithms for smooth, strongly convex problems, leading to algorithms with provable exponential convergence.

Contribution

It develops a polynomial matrix inequality approach and formulates a polynomial optimization problem, enabling the automated design of first-order algorithms with guaranteed convergence rates.

Findings

Designed a new first-order algorithm similar to Nesterov's method.

Demonstrated the framework's ability to optimize convergence rates.

Validated the approach through numerical experiments.

Abstract

In this paper, we propose a framework based on sum-of-squares programming to design iterative first-order optimization algorithms for smooth and strongly convex problems. Our starting point is to develop a polynomial matrix inequality as a sufficient condition for exponential convergence of the algorithm. The entries of this matrix are polynomial functions of the unknown parameters (exponential decay rate, stepsize, momentum coefficient, etc.). We then formulate a polynomial optimization, in which the objective is to optimize the exponential decay rate over the parameters of the algorithm. Finally, we use sum-of-squares programming as a tractable relaxation of the proposed polynomial optimization problem. We illustrate the utility of the proposed framework by designing a first-order algorithm that shares the same structure as Nesterov's accelerated gradient method.