Derivative-Free Optimization of Noisy Functions via Quasi-Newton Methods

TL;DR

This paper introduces a scalable quasi-Newton method for optimizing noisy functions, utilizing adaptive differencing and noise estimation to improve robustness and convergence in noisy settings.

Contribution

It proposes a novel finite difference quasi-Newton approach with adaptive differencing and a recovery mechanism for noisy function minimization.

Findings

The method effectively handles noisy functions with competitive performance.

The adaptive differencing improves robustness against noise.

Numerical experiments demonstrate advantages over trust region methods.

Abstract

This paper presents a finite difference quasi-Newton method for the minimization of noisy functions. The method takes advantage of the scalability and power of BFGS updating, and employs an adaptive procedure for choosing the differencing interval $h$ based on the noise estimation techniques of Hamming (2012) and Mor\'e and Wild (2011). This noise estimation procedure and the selection of $h$ are inexpensive but not always accurate, and to prevent failures the algorithm incorporates a recovery mechanism that takes appropriate action in the case when the line search procedure is unable to produce an acceptable point. A novel convergence analysis is presented that considers the effect of a noisy line search procedure. Numerical experiments comparing the method to a function interpolating trust region method are presented.