Algorithm Selection with Probing Trajectories: Benchmarking the Choice of Classifier Model

TL;DR

This paper benchmarks 17 classifiers on probing trajectories for algorithm selection, revealing that classifier choice significantly impacts performance and that feature-based models excel in this context.

Contribution

It provides the first comprehensive benchmark study comparing classifiers for trajectory-based algorithm selection using time-series data.

Findings

Feature-based and interval-based models perform best.

Classifier choice significantly affects selection accuracy.

Probing trajectories are effective for algorithm performance prediction.

Abstract

Recent approaches to training algorithm selectors in the black-box optimisation domain have advocated for the use of training data that is algorithm-centric in order to encapsulate information about how an algorithm performs on an instance, rather than relying on information derived from features of the instance itself. Probing-trajectories that consist of a sequence of objective performance per function evaluation obtained from a short run of an algorithm have recently shown particular promise in training accurate selectors. However, training models on this type of data requires an appropriately chosen classifier given the sequential nature of the data. There are currently no clear guidelines for choosing the most appropriate classifier for algorithm selection using time-series data from the plethora of models available. To address this, we conduct a large benchmark study using 17 different classifiers and three types of trajectory on a classification task using the BBOB benchmark suite using both leave-one-instance out and leave-one-problem out cross-validation. In contrast to previous studies using tabular data, we find that the choice of classifier has a significant impact, showing that feature-based and interval-based models are the best choices.