Learning Acceleration Algorithms for Fast Parametric Convex Optimization with Certified Robustness

TL;DR

This paper introduces a machine learning framework to optimize hyperparameters of accelerated first-order methods for convex optimization, achieving fast, robust solutions with minimal training data.

Contribution

It presents a novel regularization-based training approach using performance estimation problems to learn hyperparameters with certified worst-case robustness.

Findings

Significant improvement in solution quality within iteration budgets.

Strong robustness guarantees over parameter sets.

High data efficiency with only ten training instances.

Abstract

We develop a machine-learning framework to learn hyperparameter sequences for accelerated first-order methods (e.g., the step size and momentum sequences in accelerated gradient descent) to quickly solve parametric convex optimization problems with certified robustness. We obtain a strong form of robustness guarantee -- certification of worst-case performance over all parameters within a set after a given number of iterations -- through regularization-based training. The regularization term is derived from the performance estimation problem (PEP) framework based on semidefinite programming, in which the hyperparameters appear as problem data. We show how to use gradient-based training to learn the hyperparameters for several first-order methods: accelerated versions of gradient descent, proximal gradient descent, and alternating direction method of multipliers. Through various numerical examples from signal processing, control, and statistics, we demonstrate that the quality of the solution can be dramatically improved within a budget of iterations, while also maintaining strong robustness guarantees. Notably, our approach is highly data-efficient in that we only use ten training instances in all of the numerical examples.