A line search framework with restarting for noisy optimization problems

TL;DR

This paper introduces a line search optimization framework with a restarting mechanism that ensures convergence and efficiency in noisy environments, demonstrated through nonlinear conjugate gradient and quasi-Newton methods.

Contribution

It presents a novel line search framework with restart conditions that provide complexity guarantees in noisy optimization, improving robustness and practical performance.

Findings

Restarting mechanism maintains efficiency in noisy conditions.

Framework provides iteration and evaluation complexity guarantees.

Experimental results show robustness of the proposed methods.

Abstract

Nonlinear optimization methods are typically iterative and make use of gradient information to determine a direction of improvement and function information to effectively check for progress. When this information is corrupted by noise, designing a convergent and practical algorithmic process becomes challenging, as care must be taken to avoid taking bad steps due to erroneous information. For this reason, simple gradient-based schemes have been quite popular, despite being outperformed by more advanced techniques in the noiseless setting. In this paper, we propose a general algorithmic framework based on line search that is endowed with iteration and evaluation complexity guarantees even in a noisy setting. These guarantees are obtained as a result of a restarting condition, that monitors desirable properties for the steps taken at each iteration and can be checked even in the presence of noise. Experiments using a nonlinear conjugate gradient variant and a quasi-Newton variant illustrate that restarting can be performed without compromising practical efficiency and robustness.