Practical Regularized Quasi-Newton Methods with Inexact Function Values

TL;DR

This paper introduces a noise-tolerant regularized quasi-Newton method with a relaxed line search that remains stable under inexact function evaluations, improving robustness in noisy optimization scenarios.

Contribution

The paper develops a novel regularized quasi-Newton method with a relaxed line search and adaptive regularization, achieving global convergence under noisy function evaluations.

Findings

Method is more robust than existing approaches in noisy settings.

Achieves a convergence rate of O(1/ε²) for first-order stationary points.

Performs well on benchmark problems with low-precision arithmetic.

Abstract

Many practical optimization problems involve objective function values that are corrupted by unavoidable numerical errors. In smooth nonconvex optimization, quasi-Newton methods combined with line search are widely used due to their efficiency and scalability. These methods implicitly assume accurate function evaluations and thus may fail to converge in noisy settings. Developing fast and robust quasi-Newton methods for such scenarios is therefore crucial. To address this issue, we propose a noise-tolerant regularized quasi-Newton method equipped with a relaxed Armijo-type line search, designed to remain stable under inaccurate function evaluations. By combining a regularization parameter update rule inspired by Objective-Function-Free Optimization and the AdaGrad-Norm method, we establish a global convergence rate of $\mathcal{O}(1/\varepsilon^2)$ for reaching a first-order stationary point under the assumed error model. We performed extensive experiments on the CUTEst benchmark collection with artificially noisy objective function evaluations, as well as with low-precision floating-point arithmetic (64-, 32-, and 16-bit). The results demonstrate that the proposed method is substantially more robust than several existing methods, while maintaining competitive practical convergence speed and computational cost.