Estimation of partial rankings from sparse, noisy comparisons

TL;DR

This paper introduces a Bayesian approach for estimating partial rankings with ties from sparse, noisy pairwise comparison data, improving interpretability over traditional methods by only distinguishing ranks when supported by sufficient evidence.

Contribution

It proposes a nonparametric Bayesian framework and a fast algorithm for MAP inference of partial rankings, addressing limitations of existing methods in sparse, noisy data scenarios.

Findings

More parsimonious data summaries than traditional ranking methods

Effective in sparse and noisy comparison settings

Applicable to real and synthetic datasets

Abstract

Ranking items based on pairwise comparisons is common, from using match outcomes to rank sports teams to using purchase or survey data to rank consumer products. Statistical inference-based methods such as the Bradley-Terry model, which extract rankings based on an underlying generative model, have emerged as flexible and powerful tools to tackle ranking in empirical data. In situations with limited and/or noisy comparisons, it is often challenging to confidently distinguish the performance of different items based on the evidence available in the data. However, most inference-based ranking methods choose to assign each item to a unique rank or score, suggesting a meaningful distinction when there is none. Here, we develop a principled nonparametric Bayesian method, adaptable to any statistical ranking method, for learning partial rankings (rankings with ties) that distinguishes among the ranks of different items only when there is sufficient evidence available in the data. We develop a fast agglomerative algorithm to perform Maximum A Posteriori (MAP) inference of partial rankings under our framework and examine the performance of our method on a variety of real and synthetic network datasets, finding that it frequently gives a more parsimonious summary of the data than traditional ranking, particularly when observations are sparse.