Accelerated Gradient Descent by Concatenation of Stepsize Schedules

TL;DR

This paper introduces a novel technique for constructing stepsize schedules in gradient descent by concatenating simpler schedules, resulting in improved convergence rates for smooth convex optimization.

Contribution

It proposes a unified, analytic method for creating stepsize schedules through concatenation, leading to new families with optimal convergence properties.

Findings

Achieves convergence rate of O(n^{- ext{log}_2(rac{ ext{sqrt}(2)+1)}{1}})

Constructs stepsize schedules with analytic bounds for any number of iterations

Surpasses or matches the performance of existing numerically optimized schedules

Abstract

This work considers stepsize schedules for gradient descent on smooth convex objectives. We extend the existing literature and propose a unified technique for constructing stepsizes with analytic bounds for an arbitrary number of iterations. This technique constructs new stepsize schedules by concatenating two stepsize schedules with fewer steps. Using this approach, we introduce two new families of stepsize schedules, achieving a convergence rate of $O(n^{-\log_2(\sqrt 2+1)})$ with state-of-the-art constants for the objective value and gradient norm of the last iterate, respectively. Furthermore, our analytically derived stepsize schedules either match or surpass the existing best numerically computed stepsize schedules.