SELECTOR: Selecting a Representative Benchmark Suite for Reproducible Statistical Comparison

TL;DR

This paper proposes heuristics for selecting diverse, representative benchmark problem instances to improve the robustness and consistency of statistical algorithm performance comparisons in optimization.

Contribution

It introduces clustering and graph-based heuristics for selecting non-redundant, representative problem sets, enhancing reproducibility and reliability in algorithm benchmarking.

Findings

Selected benchmarks yield consistent statistical results.

Traditional benchmarks can produce conflicting algorithm performance outcomes.

Heuristics improve robustness of comparative analysis.

Abstract

Fair algorithm evaluation is conditioned on the existence of high-quality benchmark datasets that are non-redundant and are representative of typical optimization scenarios. In this paper, we evaluate three heuristics for selecting diverse problem instances which should be involved in the comparison of optimization algorithms in order to ensure robust statistical algorithm performance analysis. The first approach employs clustering to identify similar groups of problem instances and subsequent sampling from each cluster to construct new benchmarks, while the other two approaches use graph algorithms for identifying dominating and maximal independent sets of nodes. We demonstrate the applicability of the proposed heuristics by performing a statistical performance analysis of five portfolios consisting of three optimization algorithms on five of the most commonly used optimization benchmarks. The results indicate that the statistical analyses of the algorithms' performance, conducted on each benchmark separately, produce conflicting outcomes, which can be used to give a false indication of the superiority of one algorithm over another. On the other hand, when the analysis is conducted on the problem instances selected with the proposed heuristics, which uniformly cover the problem landscape, the statistical outcomes are robust and consistent.