Provably Faster Gradient Descent via Long Steps

TL;DR

This paper introduces a novel analysis technique for gradient descent that demonstrates how long steps, despite short-term increases, can lead to faster convergence in smooth convex optimization.

Contribution

It provides new convergence guarantees for gradient descent with nonconstant step sizes using a computer-assisted analysis approach.

Findings

Long steps can accelerate convergence despite short-term increases.

The analysis supports a potential $O(1/T\log T)$ convergence rate.

Numerical experiments validate the theoretical insights.

Abstract

This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique. Our theory allows nonconstant stepsize policies with frequent long steps potentially violating descent by analyzing the overall effect of many iterations at once rather than the typical one-iteration inductions used in most first-order method analyses. We show that long steps, which may increase the objective value in the short term, lead to provably faster convergence in the long term. A conjecture towards proving a faster $O(1/T\log T)$ rate for gradient descent is also motivated along with simple numerical validation.