Data-driven Analysis of First-Order Methods via Distributionally Robust Optimization

TL;DR

This paper introduces a data-driven framework combining performance estimation and distributionally robust optimization to analyze the probabilistic performance of first-order methods in convex optimization, providing less conservative guarantees.

Contribution

It unifies worst-case and average-case analyses using observed convergence data, enabling probabilistic performance guarantees through a tractable semidefinite program.

Findings

Reduces conservatism of classical worst-case bounds

Provides probabilistic guarantees based on observed data

Narrower gap between theoretical and empirical performance

Abstract

We consider the problem of analyzing the probabilistic performance of first-order methods when solving convex optimization problems drawn from an unknown distribution only accessible through samples. By combining performance estimation (PEP) and Wasserstein distributionally robust optimization (DRO), we formulate the analysis as a tractable semidefinite program. Our approach unifies worst-case and average-case analyses by incorporating data-driven information from the observed convergence of first-order methods on a limited number of problem instances. This yields probabilistic, data-driven performance guarantees in terms of the expectation or conditional value-at-risk of the selected performance metric. Experiments on smooth convex minimization, logistics regression, and Lasso show that our method significantly reduces the conservatism of classical worst-case bounds and narrows the gap between theoretical and empirical performance.