Measuring Hypothesis Testing Errors in the Evaluation of Retrieval Systems

TL;DR

This paper investigates the errors in hypothesis testing during IR system evaluation, emphasizing the importance of measuring both false positives and negatives to better assess the quality of relevance assessments.

Contribution

It introduces the quantification of Type II errors and advocates for using balanced accuracy to evaluate the discriminative power of relevance assessments in IR evaluation.

Findings

Quantifying Type II errors provides additional insights into relevance assessment quality.

Balanced accuracy effectively summarizes the discriminative power of qrels.

Using alternative relevance assessment methods impacts hypothesis testing errors.

Abstract

The evaluation of Information Retrieval (IR) systems typically uses query-document pairs with corresponding human-labelled relevance assessments (qrels). These qrels are used to determine if one system is better than another based on average retrieval performance. Acquiring large volumes of human relevance assessments is expensive. Therefore, more efficient relevance assessment approaches have been proposed, necessitating comparisons between qrels to ascertain their efficacy. Discriminative power, i.e. the ability to correctly identify significant differences between systems, is important for drawing accurate conclusions on the robustness of qrels. Previous work has measured the proportion of pairs of systems that are identified as significantly different and has quantified Type I statistical errors. Type I errors lead to incorrect conclusions due to false positive significance tests. We argue that also identifying Type II errors (false negatives) is important as they lead science in the wrong direction. We quantify Type II errors and propose that balanced classification metrics, such as balanced accuracy, can be used to portray the discriminative power of qrels. We perform experiments using qrels generated using alternative relevance assessment methods to investigate measuring hypothesis testing errors in IR evaluation. We find that additional insights into the discriminative power of qrels can be gained by quantifying Type II errors, and that balanced classification metrics can be used to give an overall summary of discriminative power in one, easily comparable, number.