Regularized Nonlinear Acceleration

TL;DR

This paper introduces a convergence acceleration technique for unconstrained optimization that computes improved solution estimates in real-time by averaging iterates, enhancing existing methods without disrupting their execution.

Contribution

It presents a novel nonlinear averaging scheme with an online-updatable linear system for convergence acceleration in optimization algorithms.

Findings

Effective acceleration demonstrated on classical classification problems.

The method runs in parallel, providing real-time improved estimates.

Numerical experiments confirm improved convergence performance.

Abstract

We describe a convergence acceleration technique for unconstrained optimization problems. Our scheme computes estimates of the optimum from a nonlinear average of the iterates produced by any optimization method. The weights in this average are computed via a simple linear system, whose solution can be updated online. This acceleration scheme runs in parallel to the base algorithm, providing improved estimates of the solution on the fly, while the original optimization method is running. Numerical experiments are detailed on classical classification problems.