A generic adaptive restart scheme with applications to saddle point algorithms

TL;DR

This paper introduces a simple, adaptive restart scheme for convex optimization that improves convergence bounds and is applicable to saddle-point algorithms, demonstrating significant practical and theoretical benefits.

Contribution

It presents the first adaptive restart algorithm for saddle-point methods like PDHG and extragradient, matching optimal bounds without prior knowledge of problem constants.

Findings

Improves worst-case bounds of PDHG on bilinear games.

Demonstrates significant practical speedups in quadratic assignment and matrix games.

Achieves near-optimal bounds for accelerated gradient descent in high-accuracy regimes.

Abstract

We provide a simple and generic adaptive restart scheme for convex optimization that is able to achieve worst-case bounds matching (up to constant multiplicative factors) optimal restart schemes that require knowledge of problem specific constants. The scheme triggers restarts whenever there is sufficient reduction of a distance-based potential function. This potential function is always computable. We apply the scheme to obtain the first adaptive restart algorithm for saddle-point algorithms including primal-dual hybrid gradient (PDHG) and extragradient. The method improves the worst-case bounds of PDHG on bilinear games, and numerical experiments on quadratic assignment problems and matrix games demonstrate dramatic improvements for obtaining high-accuracy solutions. Additionally, for accelerated gradient descent (AGD), this scheme obtains a worst-case bound within 60% of the bound achieved by the (unknown) optimal restart period when high accuracy is desired. In practice, the scheme is competitive with the heuristic of O'Donoghue and Candes (2015).