Smooth Strongly Convex Interpolation and Exact Worst-case Performance of First-order Methods

TL;DR

This paper develops a convex programming approach to exactly determine the worst-case performance of fixed-step first-order methods for smooth (strongly) convex optimization, providing explicit bounds and functions.

Contribution

It introduces a finite-dimensional semidefinite programming framework for exact worst-case analysis of first-order methods on smooth convex functions.

Findings

Derived exact worst-case bounds for various first-order methods.

Provided explicit functions matching the worst-case performance bounds.

Estimated optimal step sizes for gradient methods.

Abstract

We show that the exact worst-case performance of fixed-step first-order methods for unconstrained optimization of smooth (possibly strongly) convex functions can be obtained by solving convex programs. Finding the worst-case performance of a black-box first-order method is formulated as an optimization problem over a set of smooth (strongly) convex functions and initial conditions. We develop closed-form necessary and sufficient conditions for smooth (strongly) convex interpolation, which provide a finite representation for those functions. This allows us to reformulate the worst-case performance estimation problem as an equivalent finite dimension-independent semidefinite optimization problem, whose exact solution can be recovered up to numerical precision. Optimal solutions to this performance estimation problem provide both worst-case performance bounds and explicit functions matching them, as our smooth (strongly) convex interpolation procedure is constructive. Our works build on those of Drori and Teboulle in [Math. Prog. 145 (1-2), 2014] who introduced and solved relaxations of the performance estimation problem for smooth convex functions. We apply our approach to different fixed-step first-order methods with several performance criteria, including objective function accuracy and gradient norm. We conjecture several numerically supported worst-case bounds on the performance of the fixed-step gradient, fast gradient and optimized gradient methods, both in the smooth convex and the smooth strongly convex cases, and deduce tight estimates of the optimal step size for the gradient method.