Glocal Smoothness: Line search and adaptive step sizes can help in theory too!

TL;DR

This paper introduces a 'glocal' smoothness framework that simplifies the analysis of optimization algorithms, showing how line searches and adaptive step sizes can improve convergence rates independently of iterates.

Contribution

It proposes a new 'glocal' smoothness characterization that enables direct comparison of iteration complexities and demonstrates the benefits of line searches and adaptive steps across various algorithms.

Findings

Line searches outperform fixed step sizes under glocal smoothness.

Gradient descent with line search can have better complexity than accelerated methods.

Glocal smoothness improves complexity bounds for multiple optimization algorithms.

Abstract

Iteration complexities for optimizing smooth functions with first-order algorithms are typically stated in terms of a global Lipschitz constant of the gradient, and near-optimal results are then achieved using fixed step sizes. But many objective functions that arise in practice have regions with small Lipschitz constants where larger step sizes can be used. Many local Lipschitz assumptions have been proposed, which have led to results showing that adaptive step sizes and/or line searches yield improved convergence rates over fixed step sizes. However, these faster rates tend to depend on the iterates of the algorithm, which makes it difficult to compare the iteration complexities of different methods. We consider a simple characterization of global and local ("glocal") smoothness that only depends on properties of the function. This allows upper bounds on iteration complexities in terms of iterate-independent constants and enables us to compare iteration complexities between algorithms. Under this assumption it is straightforward to show the advantages of line searches over fixed step sizes and that, in some settings, gradient descent with line search has a better iteration complexity than accelerated methods with fixed step sizes. We further show that glocal smoothness can lead to improved complexities for the Polyak and AdGD step sizes, as well other algorithms including coordinate optimization, stochastic gradient methods, accelerated gradient methods, and non-linear conjugate gradient methods.