RankDCG: Rank-Ordering Evaluation Measure
Denys Katerenchuk, Andrew Rosenberg

TL;DR
The paper introduces rankDCG, a new ranking evaluation measure that overcomes limitations of existing metrics like nDCG, with demonstrated effectiveness on various datasets and an available software package.
Contribution
A novel ranking measure, rankDCG, that addresses key issues in existing metrics and satisfies comprehensive evaluation criteria.
Findings
rankDCG outperforms existing measures on constructed datasets
rankDCG provides consistent and unambiguous scores
The authors release an open-source evaluation package
Abstract
Ranking is used for a wide array of problems, most notably information retrieval (search). There are a number of popular approaches to the evaluation of ranking such as Kendall's , Average Precision, and nDCG. When dealing with problems such as user ranking or recommendation systems, all these measures suffer from various problems, including an inability to deal with elements of the same rank, inconsistent and ambiguous lower bound scores, and an inappropriate cost function. We propose a new measure, rankDCG, that addresses these problems. This is a modification of the popular nDCG algorithm. We provide a number of criteria for any effective ranking algorithm and show that only rankDCG satisfies all of them. Results are presented on constructed and real data sets. We release a publicly available rankDCG evaluation package.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
