A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity

TL;DR

This paper presents a simplified Lyapunov-based proof for the near-linear convergence of adaptive gradient descent on convex functions with fourth-order growth, improving understanding and performance.

Contribution

It offers a direct, simpler proof method and introduces a more adaptive algorithm with promising numerical results.

Findings

Proves near-linear convergence under convexity and fourth-order growth.

Provides a more adaptive variant of the original algorithm.

Demonstrates encouraging numerical performance.

Abstract

Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically away from their minimizers. The argument is intricate, relying on monitoring the performance of the algorithm relative to a certain manifold of slow growth -- called the ravine. In this work, we provide a direct Lyapunov-based argument that bypasses these difficulties when the objective is in addition convex and a has a unique minimizer. As a byproduct of the argument, we obtain a more adaptive variant than the original algorithm with encouraging numerical performance.