Unbiased Comparative Evaluation of Ranking Functions

TL;DR

This paper presents a unified, theoretically grounded approach to ranking evaluation using probabilistic sampling, significantly reducing the number of relevance judgments needed while maintaining unbiasedness.

Contribution

It introduces a Monte Carlo estimation framework for IR metrics, deriving optimal estimators and sampling distributions for various evaluation scenarios.

Findings

Reduces relevance judgments by at least half

Provides unbiased estimators for multiple evaluation scenarios

Demonstrates practical effectiveness over previous heuristics

Abstract

Eliciting relevance judgments for ranking evaluation is labor-intensive and costly, motivating careful selection of which documents to judge. Unlike traditional approaches that make this selection deterministically, probabilistic sampling has shown intriguing promise since it enables the design of estimators that are provably unbiased even when reusing data with missing judgments. In this paper, we first unify and extend these sampling approaches by viewing the evaluation problem as a Monte Carlo estimation task that applies to a large number of common IR metrics. Drawing on the theoretical clarity that this view offers, we tackle three practical evaluation scenarios: comparing two systems, comparing $k$ systems against a baseline, and ranking $k$ systems. For each scenario, we derive an estimator and a variance-optimizing sampling distribution while retaining the strengths of sampling-based evaluation, including unbiasedness, reusability despite missing data, and ease of use in practice. In addition to the theoretical contribution, we empirically evaluate our methods against previously used sampling heuristics and find that they generally cut the number of required relevance judgments at least in half.