Restarting Algorithms: Sometimes there is Free Lunch

TL;DR

This paper reviews the strategic use of algorithm restarts as a black-box technique to enhance performance across various optimization domains, often significantly improving convergence rates and guarantees.

Contribution

It provides a comprehensive overview of restart techniques across multiple fields, highlighting their effectiveness and versatility without requiring major modifications to base algorithms.

Findings

Restarts can improve convergence from sublinear to linear rates.

They are effective in continuous and discrete optimization problems.

Restarts are a powerful, black-box meta-technique for performance enhancement.

Abstract

In this overview article we will consider the deliberate restarting of algorithms, a meta technique, in order to improve the algorithm's performance, e.g., convergence rates or approximation guarantees. One of the major advantages is that restarts are relatively black box, not requiring any (significant) changes to the base algorithm that is restarted or the underlying argument, while leading to potentially significant improvements, e.g., from sublinear to linear rates of convergence. Restarts are widely used in different fields and have become a powerful tool to leverage additional information that has not been directly incorporated in the base algorithm or argument. We will review restarts in various settings from continuous optimization, discrete optimization, and submodular function maximization where they have delivered impressive results.