Towards Meaningful Statements in IR Evaluation. Mapping Evaluation Measures to Interval Scales
Marco Ferrante, Nicola Ferro, Norbert Fuhr

TL;DR
This paper investigates the impact of scale type on IR evaluation measures, demonstrating that converting measures to interval scales significantly alters evaluation outcomes and significance testing results.
Contribution
It introduces a method to transform IR evaluation measures into interval scales and evaluates its effects on evaluation results and significance testing.
Findings
Significant changes in measure values after scaling.
Altered significance test outcomes, with some differences becoming insignificant and vice versa.
Approximately 25% change in significance decisions across measures.
Abstract
Recently, it was shown that most popular IR measures are not interval-scaled, implying that decades of experimental IR research used potentially improper methods, which may have produced questionable results. However, it was unclear if and to what extent these findings apply to actual evaluations and this opened a debate in the community with researchers standing on opposite positions about whether this should be considered an issue (or not) and to what extent. In this paper, we first give an introduction to the representational measurement theory explaining why certain operations and significance tests are permissible only with scales of a certain level. For that, we introduce the notion of meaningfulness specifying the conditions under which the truth (or falsity) of a statement is invariant under permissible transformations of a scale. Furthermore, we show how the recall base and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
