Nonlinear Acceleration of Momentum and Primal-Dual Algorithms

TL;DR

This paper introduces nonlinear acceleration schemes for multistep optimization algorithms, enhancing convergence speed for methods including those with momentum or primal-dual structures, supported by theoretical analysis and numerical experiments.

Contribution

It presents a novel nonlinear averaging approach for acceleration that applies to non-symmetric fixed-point operators, extending acceleration techniques to a broader class of algorithms.

Findings

Acceleration performance is linked to Chebyshev problem solutions.

The method effectively accelerates Nesterov's and primal-dual algorithms.

Numerical experiments demonstrate improved convergence in logistic regression.

Abstract

We describe convergence acceleration schemes for multistep optimization algorithms. The extrapolated solution is written as a nonlinear average of the iterates produced by the original optimization method. Our analysis does not need the underlying fixed-point operator to be symmetric, hence handles e.g. algorithms with momentum terms such as Nesterov's accelerated method, or primal-dual methods. The weights are computed via a simple linear system and we analyze performance in both online and offline modes. We use Crouzeix's conjecture to show that acceleration performance is controlled by the solution of a Chebyshev problem on the numerical range of a non-symmetric operator modeling the behavior of iterates near the optimum. Numerical experiments are detailed on logistic regression problems.