Optimizing positional scoring rules for rank aggregation

TL;DR

This paper investigates how to optimally select positional scoring rules for rank aggregation in crowdsourcing, balancing theoretical complexity results with practical experiments on real and synthetic data.

Contribution

It introduces a formal optimization framework for choosing the best positional scoring rule based on partial knowledge of the true ranking, with new complexity analyses and empirical validation.

Findings

Complexity results show certain cases are computationally hard.

Proposed methods can effectively approximate the optimal scoring rule.

Experimental results demonstrate practical applicability on real-world data.

Abstract

Nowadays, several crowdsourcing projects exploit social choice methods for computing an aggregate ranking of alternatives given individual rankings provided by workers. Motivated by such systems, we consider a setting where each worker is asked to rank a fixed (small) number of alternatives and, then, a positional scoring rule is used to compute the aggregate ranking. Among the apparently infinite such rules, what is the best one to use? To answer this question, we assume that we have partial access to an underlying true ranking. Then, the important optimization problem to be solved is to compute the positional scoring rule whose outcome, when applied to the profile of individual rankings, is as close as possible to the part of the underlying true ranking we know. We study this fundamental problem from a theoretical viewpoint and present positive and negative complexity results and, furthermore, complement our theoretical findings with experiments on real-world and synthetic data.