Categorical, Ratio, and Professorial Data: The Case for Reciprocal Rank

TL;DR

This paper advocates for using reciprocal rank as a ratio-scale measure for search engine results, enabling meaningful averaging and comparison of retrieval systems beyond traditional categorical methods.

Contribution

It supports the use of reciprocal rank as a valid ratio-scale metric for SERP evaluation, challenging restrictive conditions on effectiveness mappings.

Findings

Reciprocal rank can be justified as a ratio-scale measure.

Averaging SERP scores is valid under certain effectiveness mappings.

Supports broader use of numeric metrics in search evaluation.

Abstract

Search engine results pages are usually abstracted as binary relevance vectors and hence are categorical data, meaning that only a limited set of operations is permitted, most notably tabulation of occurrence frequencies, with determination of medians and averages not possible. To compare retrieval systems it is thus usual to make use of a categorical-to-numeric effectiveness mapping. A previous paper has argued that any desired categorical-to-numeric mapping may be used, provided only that there is an argued connection between each category of SERP and the score that is assigned to that category by the mapping. Further, once that plausible connection has been established, then the mapped values can be treated as real-valued observations on a ratio scale, allowing the computation of averages. This article is written in support of that point of view, and to respond to ongoing claims that SERP scores may only be averaged if very restrictive conditions are imposed on the effectiveness mapping.