A Trust Region Method for the Optimization of Noisy Functions

TL;DR

This paper introduces a modified trust region method designed to optimize noisy functions, ensuring convergence to a neighborhood of stationarity despite errors in function evaluations.

Contribution

It proposes a simple, error-aware modification to classical trust region algorithms that requires only error size information without extra computational cost.

Findings

The algorithm converges to a neighborhood of stationarity despite noise.

Classical trust region methods may fail under noisy conditions.

Numerical results demonstrate the robustness of the proposed method.

Abstract

Classical trust region methods were designed to solve problems in which function and gradient information are exact. This paper considers the case when there are bounded errors (or noise) in the above computations and proposes a simple modification of the trust region method to cope with these errors. The new algorithm only requires information about the size of the errors in the function evaluations and incurs no additional computational expense. It is shown that, when applied to a smooth (but not necessarily convex) objective function, the iterates of the algorithm visit a neighborhood of stationarity infinitely often, and that the rest of the sequence cannot stray too far away, as measured by function values. Numerical results illustrate how the classical trust region algorithm may fail in the presence of noise, and how the proposed algorithm ensures steady progress towards stationarity in these cases.