Extensible Logging and Empirical Attainment Function for IOHexperimenter

TL;DR

This paper enhances the IOHexperimenter benchmarking platform by refactoring its logging system to be more extensible, enabling detailed performance analysis through Empirical Attainment Functions and related statistics.

Contribution

It introduces a modular logging system for IOHexperimenter and implements a new logger to compute and analyze performance metrics using Empirical Attainment Functions.

Findings

Enhanced logging system is more extensible and modular.

New logger computes generic performance views using EAF.

Provides statistical tools like Empirical Attainment Histogram.

Abstract

In order to allow for large-scale, landscape-aware, per-instance algorithm selection, a benchmarking platform software is key. IOHexperimenter provides a large set of synthetic problems, a logging system, and a fast implementation. In this work, we refactor IOHexperimenter's logging system, in order to make it more extensible and modular. Using this new system, we implement a new logger, which aims at computing performance metrics of an algorithm across a benchmark. The logger computes the most generic view on an anytime stochastic heuristic performances, in the form of the Empirical Attainment Function (EAF). We also provide some common statistics on the EAF and its discrete counterpart, the Empirical Attainment Histogram. Our work has eventually been merged in the IOHexperimenter codebase.