Full-low evaluation methods for derivative-free optimization

TL;DR

This paper introduces Full-Low Evaluation methods for derivative-free optimization that adaptively combine expensive, accurate iterations with cheap, robust ones to efficiently optimize functions of varying smoothness and noise levels.

Contribution

The paper presents a novel class of optimization methods that switch between full-evaluation and low-evaluation iterations, providing robustness and efficiency across diverse problem settings.

Findings

Methods are effective for smooth and non-smooth functions.

Switching mechanism improves robustness in noisy environments.

Theoretical complexity results match practical performance.

Abstract

We propose a new class of rigorous methods for derivative-free optimization with the aim of delivering efficient and robust numerical performance for functions of all types, from smooth to non-smooth, and under different noise regimes. To this end, we have developed Full-Low Evaluation methods, organized around two main types of iterations. The first iteration type is expensive in function evaluations, but exhibits good performance in the smooth and non-noisy cases. For the theory, we consider a line search based on an approximate gradient, backtracking until a sufficient decrease condition is satisfied. In practice, the gradient was approximated via finite differences, and the direction was calculated by a quasi-Newton step (BFGS). The second iteration type is cheap in function evaluations, yet more robust in the presence of noise or non-smoothness. For the theory, we consider direct search, and in practice we use probabilistic direct search with one random direction and its negative. A switch condition from Full-Eval to Low-Eval iterations was developed based on the values of the line-search and direct-search stepsizes. If enough Full-Eval steps are taken, we derive a complexity result of gradient-descent type. Under failure of Full-Eval, the Low-Eval iterations become the drivers of convergence yielding non-smooth convergence. Full-Low Evaluation methods are shown to be efficient and robust in practice across problems with different levels of smoothness and noise.