Adaptive Restart for Accelerated Gradient Schemes

TL;DR

This paper introduces a simple heuristic adaptive restart method for accelerated gradient schemes that significantly enhances convergence rates by resetting momentum based on observed periodic behavior, often achieving optimal convergence without prior parameter knowledge.

Contribution

The paper proposes a novel adaptive restart heuristic for accelerated gradient methods, analyzing its effectiveness in improving convergence without needing prior knowledge of function parameters.

Findings

Adaptive restart improves convergence speed.

Periodic behavior indicates when to restart momentum.

Method often recovers optimal convergence rates.

Abstract

In this paper we demonstrate a simple heuristic adaptive restart technique that can dramatically improve the convergence rate of accelerated gradient schemes. The analysis of the technique relies on the observation that these schemes exhibit two modes of behavior depending on how much momentum is applied. In what we refer to as the 'high momentum' regime the iterates generated by an accelerated gradient scheme exhibit a periodic behavior, where the period is proportional to the square root of the local condition number of the objective function. This suggests a restart technique whereby we reset the momentum whenever we observe periodic behavior. We provide analysis to show that in many cases adaptively restarting allows us to recover the optimal rate of convergence with no prior knowledge of function parameters.