Experimental Comparisons of Derivative Free Optimization Algorithms

TL;DR

This paper experimentally compares various derivative-free optimization algorithms, analyzing their performance on benchmark functions with respect to problem conditioning and rotational invariance.

Contribution

It provides a systematic evaluation of multiple derivative-free algorithms on challenging benchmark functions, highlighting their strengths and weaknesses.

Findings

Performance varies with problem conditioning.

Rotational invariance impacts algorithm effectiveness.

Different algorithms excel in different scenarios.

Abstract

In this paper, the performances of the quasi-Newton BFGS algorithm, the NEWUOA derivative free optimizer, the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), the Differential Evolution (DE) algorithm and Particle Swarm Optimizers (PSO) are compared experimentally on benchmark functions reflecting important challenges encountered in real-world optimization problems. Dependence of the performances in the conditioning of the problem and rotational invariance of the algorithms are in particular investigated.