Derivative-free optimization methods

TL;DR

This paper reviews recent advances in derivative-free optimization methods, focusing on their categorization, applications, and developments for black-box functions in scientific, engineering, and AI problems.

Contribution

It provides a comprehensive overview and unification of recent developments in derivative-free optimization, highlighting different problem settings and methodological approaches.

Findings

Categorization of methods based on function properties

Overview of deterministic and randomized approaches

Discussion of methods for stochastic and constrained problems

Abstract

In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.

Figures