Unlock the Power of Algorithm Features: A Generalization Analysis for Algorithm Selection

TL;DR

This paper provides a theoretical analysis of how incorporating algorithm features into algorithm selection models affects their generalization ability, offering bounds and insights for different learning paradigms and scenarios.

Contribution

It introduces the first provable guarantees for algorithm selection using algorithm features, analyzing their benefits, costs, and impact on generalization error under various conditions.

Findings

Derived tight upper bounds for generalization error based on Rademacher complexity.

Showed that models benefit from algorithm features in complex scenarios with many algorithms.

Established a positive correlation between generalization error and $\\chi^2$-divergence of distributions.

Abstract

In the algorithm selection research, the discussion surrounding algorithm features has been significantly overshadowed by the emphasis on problem features. Although a few empirical studies have yielded evidence regarding the effectiveness of algorithm features, the potential benefits of incorporating algorithm features into algorithm selection models and their suitability for different scenarios remain unclear. In this paper, we address this gap by proposing the first provable guarantee for algorithm selection based on algorithm features, taking a generalization perspective. We analyze the benefits and costs associated with algorithm features and investigate how the generalization error is affected by different factors. Specifically, we examine adaptive and predefined algorithm features under transductive and inductive learning paradigms, respectively, and derive upper bounds for the generalization error based on their model's Rademacher complexity. Our theoretical findings not only provide tight upper bounds, but also offer analytical insights into the impact of various factors, such as the training scale of problem instances and candidate algorithms, model parameters, feature values, and distributional differences between the training and test data. Notably, we demonstrate how models will benefit from algorithm features in complex scenarios involving many algorithms, and proves the positive correlation between generalization error bound and $\chi^2$-divergence of distributions.