Competitions in AI -- Robustly Ranking Solvers Using Statistical Resampling

TL;DR

This paper investigates the stability of AI solver competition rankings using statistical resampling, revealing their sensitivity to instance set variations and proposing a robust analysis method with confidence intervals.

Contribution

It introduces a novel resampling-based approach for statistically robust analysis of solver competition results, addressing issues of rank sensitivity and reproducibility.

Findings

Competition rankings are sensitive to small changes in benchmark sets.

Statistical ties and rank inversions are common in current competition results.

The proposed method provides confidence intervals and more reliable solver rankings.

Abstract

Solver competitions play a prominent role in assessing and advancing the state of the art for solving many problems in AI and beyond. Notably, in many areas of AI, competitions have had substantial impact in guiding research and applications for many years, and for a solver to be ranked highly in a competition carries considerable weight. But to which extent can we expect competition results to generalise to sets of problem instances different from those used in a particular competition? This is the question we investigate here, using statistical resampling techniques. We show that the rankings resulting from the standard interpretation of competition results can be very sensitive to even minor changes in the benchmark instance set used as the basis for assessment and can therefore not be expected to carry over to other samples from the same underlying instance distribution. To address this problem, we introduce a novel approach to statistically meaningful analysis of competition results based on resampling performance data. Our approach produces confidence intervals of competition scores as well as statistically robust solver rankings with bounded error. Applied to recent SAT, AI planning and computer vision competitions, our analysis reveals frequent statistical ties in solver performance as well as some inversions of ranks compared to the official results based on simple scoring.