Near Optimal Inference for the Best-Performing Algorithm

TL;DR

This paper introduces a new framework for selecting the best-performing machine learning algorithm based on benchmark data, with methods that outperform existing approaches and include theoretical performance bounds.

Contribution

It proposes a novel subset selection framework for identifying top algorithms with high confidence, improving upon existing methods with both asymptotic and finite-sample guarantees.

Findings

New subset selection methods outperform current techniques

Matching lower bounds demonstrate optimality of proposed schemes

Framework applicable to identifying most frequent symbols in data

Abstract

Consider a collection of competing machine learning algorithms. Given their performance on a benchmark of datasets, we would like to identify the best performing algorithm. Specifically, which algorithm is most likely to rank highest on a future, unseen dataset. A natural approach is to select the algorithm that demonstrates the best performance on the benchmark. However, in many cases the performance differences are marginal and additional candidates may also be considered. This problem is formulated as subset selection for multinomial distributions. Formally, given a sample from a countable alphabet, our goal is to identify a minimal subset of symbols that includes the most frequent symbol in the population with high confidence. In this work, we introduce a novel framework for the subset selection problem. We provide both asymptotic and finite-sample schemes that significantly improve upon currently known methods. In addition, we provide matching lower bounds, demonstrating the favorable performance of our proposed schemes.