Towards Reliable Testing for Multiple Information Retrieval System Comparisons

TL;DR

This paper evaluates the reliability of multiple comparison procedures in Information Retrieval system testing, proposing that Wilcoxon with Benjamini-Hochberg correction offers optimal balance between error control and statistical power.

Contribution

It introduces a new approach to assess multiple comparison procedures in IR, demonstrating the effectiveness of Wilcoxon with Benjamini-Hochberg correction through experiments.

Findings

Wilcoxon plus Benjamini-Hochberg maintains proper Type I error rates.

This combination provides the best statistical power among tested methods.

The approach is validated on simulated and real TREC data.

Abstract

Null Hypothesis Significance Testing is the \textit{de facto} tool for assessing effectiveness differences between Information Retrieval systems. Researchers use statistical tests to check whether those differences will generalise to online settings or are just due to the samples observed in the laboratory. Much work has been devoted to studying which test is the most reliable when comparing a pair of systems, but most of the IR real-world experiments involve more than two. In the multiple comparisons scenario, testing several systems simultaneously may inflate the errors committed by the tests. In this paper, we use a new approach to assess the reliability of multiple comparison procedures using simulated and real TREC data. Experiments show that Wilcoxon plus the Benjamini-Hochberg correction yields Type I error rates according to the significance level for typical sample sizes while being the best test in terms of statistical power.