Convergence of a class of gradient-free optimisation schemes when the objective function is noisy, irregular, or both

TL;DR

This paper analyzes the convergence of gradient-free optimization algorithms for noisy, irregular, and non-smooth functions, including model-based and mollification methods, with applications in machine learning.

Contribution

It provides convergence guarantees under weak regularity assumptions for a broad class of zero-th order optimization algorithms, including stochastic cases.

Findings

Convergence proven under weak regularity assumptions

Includes model-based and mollification methods as special cases

Validated through a machine learning classification example

Abstract

We investigate the convergence properties of a class of iterative algorithms designed to minimize a potentially non-smooth and noisy objective function, which may be algebraically intractable and whose values may be obtained as the output of a black box. The algorithms considered can be cast under the umbrella of a generalised gradient descent recursion, where the gradient is that of a smooth approximation of the objective function. The framework we develop includes as special cases model-based and mollification methods, two classical approaches to zero-th order optimisation. The convergence results are obtained under very weak assumptions on the regularity of the objective function and involve a trade-off between the degree of smoothing and size of the steps taken in the parameter updates. As expected, additional assumptions are required in the stochastic case. We illustrate the relevance of these algorithms and our convergence results through a challenging classification example from machine learning.