Open Problem: Anytime Convergence Rate of Gradient Descent

TL;DR

This paper investigates whether a stepsize schedule exists for gradient descent that guarantees accelerated convergence rates at any stopping time, addressing limitations of existing acceleration methods for smooth convex functions.

Contribution

The paper raises a fundamental open problem about the existence of stepsize schedules that ensure consistent acceleration in gradient descent for convex objectives.

Findings

Identifies potential for large errors with certain stepsize sequences

Highlights the open problem of achieving universal acceleration

Questions the limits of current acceleration techniques

Abstract

Recent results show that vanilla gradient descent can be accelerated for smooth convex objectives, merely by changing the stepsize sequence. We show that this can lead to surprisingly large errors indefinitely, and therefore ask: Is there any stepsize schedule for gradient descent that accelerates the classic $\mathcal{O}(1/T)$ convergence rate, at \emph{any} stopping time $T$?