An optimal first order method based on optimal quadratic averaging

TL;DR

This paper introduces a new optimal first order method for smooth strongly convex functions based on averaging quadratic lower models, connecting geometric descent with classical optimization techniques and offering improved limited-memory variants.

Contribution

It presents a novel perspective that the optimal quadratic averaging scheme generates the same iterates as a geometric descent method, enhancing understanding and extending to limited-memory versions.

Findings

Achieves optimal convergence rate for smooth strongly convex functions

Provides a new interpretation linking geometric descent and quadratic averaging

Develops limited-memory extensions with improved performance

Abstract

In a recent paper, Bubeck, Lee, and Singh introduced a new first order method for minimizing smooth strongly convex functions. Their geometric descent algorithm, largely inspired by the ellipsoid method, enjoys the optimal linear rate of convergence. We show that the same iterate sequence is generated by a scheme that in each iteration computes an optimal average of quadratic lower-models of the function. Indeed, the minimum of the averaged quadratic approaches the true minimum at an optimal rate. This intuitive viewpoint reveals clear connections to the original fast-gradient methods and cutting plane ideas, and leads to limited-memory extensions with improved performance.