A Simple Nearly-Optimal Restart Scheme For Speeding-Up First Order Methods

TL;DR

This paper introduces a straightforward restart scheme for first-order convex optimization methods that improves efficiency without needing problem-specific parameters, achieving near-optimal complexity bounds.

Contribution

The scheme is simple, universal, and does not require problem-specific information, providing nearly optimal complexity bounds for various first-order methods.

Findings

Scheme achieves near-optimal complexity bounds.

No need for problem-specific parameter tuning.

Applicable to a wide class of convex problems.

Abstract

We present a simple scheme for restarting first-order methods for convex optimization problems. Restarts are made based only on achieving specified decreases in objective values, the specified amounts being the same for all optimization problems. Unlike existing restart schemes, the scheme makes no attempt to learn parameter values characterizing the structure of an optimization problem, nor does it require any special information that would not be available in practice (unless the first-order method chosen to be employed in the scheme itself requires special information). As immediate corollaries to the main theorems, we show that when some well-known first-order methods are employed in the scheme, the resulting complexity bounds are nearly optimal for particular -- yet quite general -- classes of problems.