Adaptive restart of accelerated gradient methods under local quadratic growth condition

TL;DR

This paper demonstrates that restarting accelerated gradient methods at any frequency under local quadratic growth guarantees linear convergence, and proposes an adaptive scheme to optimize restart frequency for improved efficiency.

Contribution

It introduces an adaptive restart scheme for accelerated gradient methods that automatically adjusts restart frequency based on observed gradient norm decreases.

Findings

The adaptive scheme outperforms previous methods in convergence speed.

Restarting at any frequency under local quadratic growth ensures linear convergence.

The method is effective on Lasso and logistic regression problems.

Abstract

By analyzing accelerated proximal gradient methods under a local quadratic growth condition, we show that restarting these algorithms at any frequency gives a globally linearly convergent algorithm. This result was previously known only for long enough frequencies. Then, as the rate of convergence depends on the match between the frequency and the quadratic error bound, we design a scheme to automatically adapt the frequency of restart from the observed decrease of the norm of the gradient mapping. Our algorithm has a better theoretical bound than previously proposed methods for the adaptation to the quadratic error bound of the objective. We illustrate the efficiency of the algorithm on a Lasso problem and on a regularized logistic regression problem.