The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong

TL;DR

This paper critically examines common evaluation practices in algorithm selection for black-box optimization, highlighting methodological flaws that can lead to misleading performance assessments of meta-models.

Contribution

It identifies specific flaws in evaluation methods like leave-instance-out and metrics sensitive to scale, proposing the need for more rigorous assessment frameworks.

Findings

Leave-instance-out evaluation can be misleading

Scale-sensitive metrics can overestimate performance

Non-informative features can inflate accuracy

Abstract

Algorithm selection, aiming to identify the best algorithm for a given problem, plays a pivotal role in continuous black-box optimization. A common approach involves representing optimization functions using a set of features, which are then used to train a machine learning meta-model for selecting suitable algorithms. Various approaches have demonstrated the effectiveness of these algorithm selection meta-models. However, not all evaluation approaches are equally valid for assessing the performance of meta-models. We highlight methodological issues that frequently occur in the community and should be addressed when evaluating algorithm selection approaches. First, we identify flaws with the "leave-instance-out" evaluation technique. We show that non-informative features and meta-models can achieve high accuracy, which should not be the case with a well-designed evaluation framework. Second, we demonstrate that measuring the performance of optimization algorithms with metrics sensitive to the scale of the objective function requires careful consideration of how this impacts the construction of the meta-model, its predictions, and the model's error. Such metrics can falsely present overly optimistic performance assessments of the meta-models. This paper emphasizes the importance of careful evaluation, as loosely defined methodologies can mislead researchers, divert efforts, and introduce noise into the field