A Geometric Structure of Acceleration and Its Role in Making Gradients Small Fast

TL;DR

This paper uncovers a geometric structure underlying many accelerated optimization methods, leading to novel algorithms that achieve faster convergence rates in reducing gradient norms.

Contribution

It introduces a unifying geometric framework for accelerated methods and proposes new algorithms with improved convergence rates.

Findings

Proposes a geometric structure common to many accelerated methods.

Develops a new accelerated method with $ ext{O}(1/K^4)$ rate.

Demonstrates faster reduction of squared gradient norm.

Abstract

Since Nesterov's seminal 1983 work, many accelerated first-order optimization methods have been proposed, but their analyses lacks a common unifying structure. In this work, we identify a geometric structure satisfied by a wide range of first-order accelerated methods. Using this geometric insight, we present several novel generalizations of accelerated methods. Most interesting among them is a method that reduces the squared gradient norm with $\mathcal{O}(1/K^4)$ rate in the prox-grad setup, faster than the $\mathcal{O}(1/K^3)$ rates of Nesterov's FGM or Kim and Fessler's FPGM-m.