Optimized first-order methods for smooth convex minimization

TL;DR

This paper develops new optimized first-order algorithms for smooth convex minimization that outperform Nesterov's methods in convergence rate and are computationally efficient for large-scale problems.

Contribution

The paper introduces analytically optimized first-order methods with improved convergence bounds and recursive structures similar to Nesterov's methods.

Findings

Achieves convergence bound twice as small as Nesterov's fast gradient methods.

Provides analytically derived optimization bounds that refine previous numerical results.

Develops recursive algorithms suitable for large-scale optimization.

Abstract

We introduce new optimized first-order methods for smooth unconstrained convex minimization. Drori and Teboulle recently described a numerical method for computing the $N$-iteration optimal step coefficients in a class of first-order algorithms that includes gradient methods, heavy-ball methods, and Nesterov's fast gradient methods. However, Drori and Teboulle's numerical method is computationally expensive for large $N$, and the corresponding numerically optimized first-order algorithm requires impractical memory and computation for large-scale optimization problems. In this paper, we propose optimized first-order algorithms that achieve a convergence bound that is two times smaller than for Nesterov's fast gradient methods; our bound is found analytically and refines the numerical bound. Furthermore, the proposed optimized first-order methods have efficient recursive forms that are remarkably similar to Nesterov's fast gradient methods.