A Noise-Tolerant Quasi-Newton Algorithm for Unconstrained Optimization

TL;DR

This paper introduces a noise-tolerant quasi-Newton algorithm that extends BFGS and L-BFGS methods to handle noisy, low-precision, or statistical data, ensuring stable updates and convergence guarantees.

Contribution

It proposes a novel lengthening procedure and a new error-tolerant line search to improve BFGS and L-BFGS methods under noisy conditions, with proven convergence for strongly convex functions.

Findings

The method maintains stability in noisy environments.

Convergence is guaranteed for strongly convex functions.

Numerical results show improved robustness and performance.

Abstract

This paper describes an extension of the BFGS and L-BFGS methods for the minimization of a nonlinear function subject to errors. This work is motivated by applications that contain computational noise, employ low-precision arithmetic, or are subject to statistical noise. The classical BFGS and L-BFGS methods can fail in such circumstances because the updating procedure can be corrupted and the line search can behave erratically. The proposed method addresses these difficulties and ensures that the BFGS update is stable by employing a lengthening procedure that spaces out the points at which gradient differences are collected. A new line search, designed to tolerate errors, guarantees that the Armijo-Wolfe conditions are satisfied under most reasonable conditions, and works in conjunction with the lengthening procedure. The proposed methods are shown to enjoy convergence guarantees for strongly convex functions. Detailed implementations of the methods are presented, together with encouraging numerical results.