Batch Evaluation Metrics in Information Retrieval: Measures, Scales, and Meaning

TL;DR

This paper defends the validity of existing IR evaluation metrics like reciprocal rank and average precision, arguing they are meaningful as ratio-scale measures of usefulness, countering recent claims that only uniform-step interval scales are appropriate.

Contribution

The paper challenges the recent push for interval scales in IR evaluation, defending the use of traditional metrics as valid ratio-scale measures of usefulness.

Findings

IR document rankings are categorical data

Effectiveness metrics can be represented as ratio-scale usefulness measures

Current IR metrics are more meaningful in their original form than intervalized versions

Abstract

A sequence of recent papers has considered the role of measurement scales in information retrieval (IR) experimentation, and presented the argument that (only) uniform-step interval scales should be used, and hence that well-known metrics such as reciprocal rank, expected reciprocal rank, normalized discounted cumulative gain, and average precision, should be either discarded as measurement tools, or adapted so that their metric values lie at uniformly-spaced points on the number line. These papers paint a rather bleak picture of past decades of IR evaluation, at odds with the community's overall emphasis on practical experimentation and measurable improvement. Our purpose in this work is to challenge that position. In particular, we argue that mappings from categorical and ordinal data to sets of points on the number line are valid provided there is an external reason for each target point to have been selected. We first consider the general role of measurement scales, and of categorical, ordinal, interval, ratio, and absolute data collections. In connection with the first two of those categories we also provide examples of the knowledge that is captured and represented by numeric mappings to the real number line. Focusing then on information retrieval, we argue that document rankings are categorical data, and that the role of an effectiveness metric is to provide a single value that represents the usefulness to a user or population of users of any given ranking, with usefulness able to be represented as a continuous variable on a ratio scale. That is, we argue that current IR metrics are well-founded, and, moreover, that those metrics are more meaningful in their current form than in the proposed "intervalized" versions.