Anytime Acceleration of Gradient Descent

TL;DR

This paper introduces a stepsize schedule for gradient descent that guarantees convergence rates that improve over time without prior knowledge of the stopping point, addressing an open problem in optimization.

Contribution

It proposes a novel stepsize schedule that achieves anytime convergence guarantees for both smooth convex and strongly convex optimization, advancing the understanding of acceleration methods.

Findings

Achieves $O(T^{-1.119})$ convergence for smooth convex functions.

Provides exponential convergence guarantees for strongly convex functions.

Answers an open problem on stepsize-based acceleration with anytime guarantees.

Abstract

This work investigates stepsize-based acceleration of gradient descent with {\em anytime} convergence guarantees. For smooth (non-strongly) convex optimization, we propose a stepsize schedule that allows gradient descent to achieve convergence guarantees of $O(T^{-1.119})$ for any stopping time $T$, where the stepsize schedule is predetermined without prior knowledge of the stopping time. This result provides an affirmative answer to a COLT open problem \citep{kornowski2024open} regarding whether stepsize-based acceleration can yield anytime convergence rates of $o(T^{-1})$. We further extend our theory to yield anytime convergence guarantees of $\exp(-\Omega(T/\kappa^{0.893}))$ for smooth and strongly convex optimization, with $\kappa$ being the condition number.