On the Influence of the Feature Computation Budget on Per-Instance Algorithm Selection for Black-Box Optimization

TL;DR

This study investigates how the allocation of the optimization budget to feature computation affects the effectiveness of per-instance algorithm selection in black-box optimization, finding that a significant portion can be spent on features without losing viability.

Contribution

It provides a comprehensive analysis of the optimal feature computation budget in PIAS for black-box optimization, considering various scenarios and tradeoffs.

Findings

PIAS remains effective even when up to 25% of the budget is used for feature computation.

The optimal feature budget varies depending on the specific algorithm selection scenario.

On average, 20% of PIAS performance loss is due to feature computation budget, emphasizing its importance.

Abstract

Per-instance algorithm selection (PIAS) takes advantage of complementarity between a set of algorithms by deciding which algorithm to run on a given instance. This decision is based on features of the instances, which, in the context of black-box optimization (BBO), require a part of the optimization budget to be computed. This raises two questions: (a) from which fraction of the budget spent on feature computation does PIAS become worth it for BBO, and (b) which fraction of the budget optimizes the tradeoff between feature accuracy and PIAS performance. To this end, we perform a broad study where PIAS with varying sampling budgets for feature computation is compared to the single best algorithm on a broad range of algorithm selection scenarios. These scenarios consist of two portfolio sizes, three problem sets, 4 dimensionalities, and 10 target budgets. We find that PIAS is viable for the majority of tested scenarios, even when as much as a quarter of the total budget is spent on feature computation. The tradeoff for the fraction of the budget spent on feature computation to maximize the benefit of PIAS is highly dependent on the specific AS scenario. Further, on average 20 percent of PIAS loss to the virtual best solver is explained by the budget spent on feature computation, highlighting the importance of properly accounting for the feature budget.