Frugal Algorithm Selection

TL;DR

This paper investigates methods to reduce the training cost of automated algorithm selection by choosing optimal training subsets and employing active learning, timeout predictors, and progressive data collection, demonstrating significant cost reductions.

Contribution

It introduces novel strategies for selecting training instances in algorithm selection, including active learning and timeout-based data collection, to lower training costs.

Findings

Active learning reduces training data labeling costs.

Timeout predictors improve algorithm performance estimation.

Progressive timeout data collection further decreases training effort.

Abstract

When solving decision and optimisation problems, many competing algorithms (model and solver choices) have complementary strengths. Typically, there is no single algorithm that works well for all instances of a problem. Automated algorithm selection has been shown to work very well for choosing a suitable algorithm for a given instance. However, the cost of training can be prohibitively large due to running candidate algorithms on a representative set of training instances. In this work, we explore reducing this cost by choosing a subset of the training instances on which to train. We approach this problem in three ways: using active learning to decide based on prediction uncertainty, augmenting the algorithm predictors with a timeout predictor, and collecting training data using a progressively increasing timeout. We evaluate combinations of these approaches on six datasets from ASLib and present the reduction in labelling cost achieved by each option.