Factor-$\sqrt{2}$ Acceleration of Accelerated Gradient Methods

TL;DR

This paper provides a human-understandable analysis of the optimized gradient method (OGM), revealing its factor-$\sqrt{2}$ acceleration in convex, strongly convex, and mirror descent settings, and generalizes its acceleration mechanism.

Contribution

The paper introduces a new human-readable analysis of OGM using Lyapunov functions and linear coupling, and extends its acceleration benefits to multiple optimization setups.

Findings

OGM achieves a factor-$\sqrt{2}$ speedup over Nesterov's method.

The analysis is based on Lyapunov functions and linear coupling, without computer assistance.

The acceleration mechanism is generalized to other optimization scenarios.

Abstract

The optimized gradient method (OGM) provides a factor-$\sqrt{2}$ speedup upon Nesterov's celebrated accelerated gradient method in the convex (but non-strongly convex) setup. However, this improved acceleration mechanism has not been well understood; prior analyses of OGM relied on a computer-assisted proof methodology, so the proofs were opaque for humans despite being verifiable and correct. In this work, we present a new analysis of OGM based on a Lyapunov function and linear coupling. These analyses are developed and presented without the assistance of computers and are understandable by humans. Furthermore, we generalize OGM's acceleration mechanism and obtain a factor-$\sqrt{2}$ speedup in other setups: acceleration with a simpler rational stepsize, the strongly convex setup, and the mirror descent setup.