Data-Driven Relevance Judgments for Ranking Evaluation

TL;DR

This paper introduces a data-driven method to improve ranking evaluation metrics by accounting for score divergence, leading to more accurate relevance judgments and better ranking assessments.

Contribution

It proposes a novel relevance function based on score divergence, enhancing the accuracy of ranking evaluation metrics like nDCG.

Findings

The proposed method provides more fine-grained ranking evaluations.

Standard nDCG often under- or over-estimates scores based on score divergence.

Synthetic and real-world data demonstrate the improved performance of the new approach.

Abstract

Ranking evaluation metrics are a fundamental element of design and improvement efforts in information retrieval. We observe that most popular metrics disregard information portrayed in the scores used to derive rankings, when available. This may pose a numerical scaling problem, causing an under- or over-estimation of the evaluation depending on the degree of divergence between the scores of ranked items. The purpose of this work is to propose a principled way of quantifying multi-graded relevance judgments of items and enable a more accurate penalization of ordering errors in rankings. We propose a data-driven generation of relevance functions based on the degree of the divergence amongst a set of items' scores and its application in the evaluation metric Normalized Discounted Cumulative Gain (nDCG). We use synthetic data to demonstrate the interest of our proposal and a combination of data on news items from Google News and their respective popularity in Twitter to show its performance in comparison to the standard nDCG. Results show that our proposal is capable of providing a more fine-grained evaluation of rankings when compared to the standard nDCG, and that the latter frequently under- or over-estimates its evaluation scores in light of the divergence of items' scores.