Refined bounds for algorithm configuration: The knife-edge of dual class approximability

TL;DR

This paper establishes refined theoretical bounds on the number of training instances needed for reliable algorithm configuration, demonstrating that the complexity depends on the norm used for performance approximation and providing empirical validation in integer programming.

Contribution

It introduces new sample complexity bounds for algorithm configuration based on performance approximation norms, improving understanding of training set size requirements.

Findings

Sample complexity bounds are significantly improved, up to 700 times smaller.

Performance approximation under L-infinity norm yields strong bounds.

Approximation under L-p norms for p<infinity does not guarantee meaningful bounds.

Abstract

Automating algorithm configuration is growing increasingly necessary as algorithms come with more and more tunable parameters. It is common to tune parameters using machine learning, optimizing performance metrics such as runtime and solution quality. The training set consists of problem instances from the specific domain at hand. We investigate a fundamental question about these techniques: how large should the training set be to ensure that a parameter's average empirical performance over the training set is close to its expected, future performance? We answer this question for algorithm configuration problems that exhibit a widely-applicable structure: the algorithm's performance as a function of its parameters can be approximated by a "simple" function. We show that if this approximation holds under the L-infinity norm, we can provide strong sample complexity bounds. On the flip side, if the approximation holds only under the L-p norm for p smaller than infinity, it is not possible to provide meaningful sample complexity bounds in the worst case. We empirically evaluate our bounds in the context of integer programming, one of the most powerful tools in computer science. Via experiments, we obtain sample complexity bounds that are up to 700 times smaller than the previously best-known bounds.