Ranking with Confidence for Large Scale Comparison Data

TL;DR

This paper introduces PD-Rank, a fast and accurate ranking algorithm from noisy pairwise comparison data that provides confidence measures and outperforms existing methods in speed and accuracy.

Contribution

We develop a novel ranking algorithm based on a generative noise model, using an efficient optimization approach that improves accuracy and scalability in large-scale comparison data.

Findings

PD-Rank achieves higher Kendall tau scores than competing methods.

It is faster by an order of magnitude compared to existing algorithms.

Performs well even with 10% incorrect comparisons in simulated data.

Abstract

In this work, we leverage a generative data model considering comparison noise to develop a fast, precise, and informative ranking algorithm from pairwise comparisons that produces a measure of confidence on each comparison. The problem of ranking a large number of items from noisy and sparse pairwise comparison data arises in diverse applications, like ranking players in online games, document retrieval or ranking human perceptions. Although different algorithms are available, we need fast, large-scale algorithms whose accuracy degrades gracefully when the number of comparisons is too small. Fitting our proposed model entails solving a non-convex optimization problem, which we tightly approximate by a sum of quasi-convex functions and a regularization term. Resorting to an iterative reweighted minimization and the Primal-Dual Hybrid Gradient method, we obtain PD-Rank, achieving a Kendall tau 0.1 higher than all comparing methods, even for 10\% of wrong comparisons in simulated data matching our data model, and leading in accuracy if data is generated according to the Bradley-Terry model, in both cases faster by one order of magnitude, in seconds. In real data, PD-Rank requires less computational time to achieve the same Kendall tau than active learning methods.