On the Metric Properties of IR Evaluation Measures Based on Ranking Axioms

TL;DR

This paper investigates the metric properties of IR evaluation measures based on ranking axioms, revealing which measures are true metrics or pseudo-metrics under certain conditions using lattice theory.

Contribution

It formally analyzes the metric properties of ranking-based IR measures using lattice theory, clarifying when they are metrics or pseudo-metrics.

Findings

Precision, recall, RBP, and DCG are metrics when relevant documents are prioritized.

These measures become pseudo-metrics when document swapping is considered.

The study provides a formal framework for understanding IR measure properties.

Abstract

The axiomatic analysis of IR evaluation metrics has contributed to a better understanding of their properties. Some works have modelled the effectiveness of retrieval measures with axioms that capture desirable properties on the set of ranked lists of documents. Recently, it has been shown that three of these axioms lead to some orderings. This work formally explores the metric properties of the set of rankings, endowed with these orderings. Based on lattice theory, the possible metrics and pseudo-metrics, defined on these structures, are determined. It is found that, when the relevant documents are prioritized, precision, recall, RBP and DCG are metrics on the set of rankings, however they are pseudo-metrics when the swapping of documents is considered.