Who is the Winning Algorithm? Rank Aggregation for Comparative Studies

TL;DR

This paper introduces a new framework for estimating the likelihood of each algorithm winning on unseen datasets by leveraging complete ranking data, outperforming existing methods in synthetic and real-world tests.

Contribution

It proposes a novel conceptual framework that utilizes full ranking information to better estimate algorithm win probabilities, advancing beyond traditional win count methods.

Findings

Significant improvement over existing methods in synthetic data.

Enhanced accuracy in real-world benchmark evaluations.

Framework effectively leverages complete ranking data.

Abstract

Consider a collection of m competing machine learning algorithms. Given their performance on a benchmark of datasets, we would like to identify the best performing algorithm. Specifically, which algorithm is most likely to ``win'' (rank highest) on a future, unseen dataset. The standard maximum likelihood approach suggests counting the number of wins per each algorithm. In this work, we argue that there is much more information in the complete rankings. That is, the number of times that each algorithm finished second, third and so forth. Yet, it is not entirely clear how to effectively utilize this information for our purpose. In this work we introduce a novel conceptual framework for estimating the win probability for each of the m algorithms, given their complete rankings over a benchmark of datasets. Our proposed framework significantly improves upon currently known methods in synthetic and real-world examples.