Alternating cyclic extrapolation methods for optimization algorithms

TL;DR

This paper presents novel alternating cyclic extrapolation methods to accelerate fixed-point iterations, offering efficient, problem-agnostic solutions suitable for high-dimensional optimization with minimal computational overhead.

Contribution

Introduces new alternating cyclic extrapolation techniques with a novel step length, applicable to various nonlinear problems and demonstrated through multiple optimization applications.

Findings

Methods outperform or match existing algorithms in efficiency.

Require minimal function evaluations and no matrix inversions.

Effective in high-dimensional optimization contexts.

Abstract

This article introduces new acceleration methods for fixed-point iterations. Extrapolations are computed using two or three mappings alternately and a new type of step length is proposed with good properties for nonlinear applications. The methods require no problem-specific adaptation and are especially efficient in high-dimensional contexts. Their computation uses few objective function evaluations, no matrix inversion and little extra memory. A convergence analysis is followed by eight applications including gradient descent acceleration for constrained and unconstrained optimization. Performances are on par with or better than competitive alternatives. The algorithm is available as the Julia package SpeedMapping.jl.