A New Correlation Coefficient for Aggregating Non-strict and Incomplete Rankings

TL;DR

This paper introduces a new correlation coefficient for aggregating incomplete and tied rankings, generalizing Kendall tau, with proven metric properties and practical optimization algorithms, improving consensus ranking accuracy especially in noisy data.

Contribution

The paper proposes a novel correlation coefficient for non-strict, incomplete rankings, establishing its theoretical properties, and developing exact optimization methods for consensus ranking.

Findings

The new coefficient satisfies metric-like axioms and aligns with Kemeny aggregation.

The algorithms perform better with noisier data, reducing alternative solutions.

Experimental results show rankings are closer to the true consensus in noisy scenarios.

Abstract

We introduce a correlation coefficient that is designed to deal with a variety of ranking formats including those containing non-strict (i.e., with-ties) and incomplete (i.e., unknown) preferences. The correlation coefficient is designed to enforce a neutral treatment of incompleteness whereby no assumptions are made about individual preferences involving unranked objects. The new measure, which can be regarded as a generalization of the seminal Kendall tau correlation coefficient, is proven to satisfy a set of metric-like axioms and to be equivalent to a recently developed ranking distance function associated with Kemeny aggregation. In an effort to further unify and enhance both robust ranking methodologies, this work proves the equivalence of an additional distance and correlation-coefficient pairing in the space of non-strict incomplete rankings. These connections induce new exact optimization methodologies: a specialized branch and bound algorithm and an exact integer programming formulation. Moreover, the bridging of these complementary theories reinforces the singular suitability of the featured correlation coefficient to solve the general consensus ranking problem. The latter premise is bolstered by an accompanying set of experiments on random instances, which are generated via a herein developed sampling technique connected with the classic Mallows distribution of ranking data. Associated experiments with the branch and bound algorithm demonstrate that, as data becomes noisier, the featured correlation coefficient yields relatively fewer alternative optimal solutions and that the aggregate rankings tend to be closer to an underlying ground truth shared by a majority.