Using the Empirical Attainment Function for Analyzing Single-objective Black-box Optimization Algorithms

TL;DR

This paper introduces the empirical attainment function (EAF) as a superior alternative to the target-based ECDF for analyzing black-box optimization algorithms, offering more precise performance insights and additional analysis tools.

Contribution

It demonstrates the advantages of EAF over ECDF, integrates EAF computation into IOHanalyzer, and illustrates its use with synthetic and BBOB benchmark data.

Findings

EAF provides more precise performance differences.

EAF captures performance without predefined quality targets.

Average area over convergence curves is an effective performance measure.

Abstract

A widely accepted way to assess the performance of iterative black-box optimizers is to analyze their empirical cumulative distribution function (ECDF) of pre-defined quality targets achieved not later than a given runtime. In this work, we consider an alternative approach, based on the empirical attainment function (EAF) and we show that the target-based ECDF is an approximation of the EAF. We argue that the EAF has several advantages over the target-based ECDF. In particular, it does not require defining a priori quality targets per function, captures performance differences more precisely, and enables the use of additional summary statistics that enrich the analysis. We also show that the average area over the convergence curves is a simpler-to-calculate, but equivalent, measure of anytime performance. To facilitate the accessibility of the EAF, we integrate a module to compute it into the IOHanalyzer platform. Finally, we illustrate the use of the EAF via synthetic examples and via the data available for the BBOB suite.