On the Effect of Ranking Axioms on IR Evaluation Metrics

TL;DR

This paper investigates how ranking axioms influence the numerical values of IR evaluation metrics, revealing structural properties and limitations in their computation based on axiomatic assumptions.

Contribution

It formally analyzes the impact of ranking axioms on IR metrics and introduces the concept of join-irreducible elements in ranking sets, enhancing understanding of metric behavior.

Findings

Ranking axioms shape the orderings of document rankings.

Precision, recall, RBP, and DCG can be derived from join-irreducible elements.

Swapping documents affects the ability to compute metrics from these elements.

Abstract

The study of IR evaluation metrics through axiomatic analysis enables a better understanding of their numerical properties. Some works have modelled the effectiveness of retrieval metrics with axioms that capture desirable properties on the set of rankings of documents. This paper formally explores the effect of these ranking axioms on the numerical values of some IR evaluation metrics. It focuses on the set of ranked lists of documents with multigrade relevance. The possible orderings in this set are derived from three commonly accepted ranking axioms on retrieval metrics; then, they are classified by their latticial properties. When relevant documents are prioritised, a subset of document rankings are identified: the join-irreducible elements, which have some resemblance to the concept of basis in vector space. It is possible to compute the precision, recall, RBP or DCG values of any ranking from their values in the join-irreducible elements. However this is not the case when the swapping of documents is considered.