Ranking a Set of Objects using Heterogeneous Workers: QUITE an Easy Problem

TL;DR

This paper introduces QUITE, a non-adaptive algorithm for ranking objects based on noisy pairwise comparisons from workers with varying reliability, jointly estimating worker reliability and object qualities.

Contribution

The paper presents QUITE, a novel non-adaptive ranking algorithm that accounts for heterogeneous worker reliabilities and object qualities, improving ranking accuracy.

Findings

QUITE outperforms previous algorithms in various scenarios.

QUITE effectively estimates worker reliabilities and object qualities.

The method can be extended to an adaptive version.

Abstract

We focus on the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of unequal workers, each worker being characterized by a specific degree of reliability, which reflects her ability to rank pairs of objects. More specifically, we assume that objects are endowed with intrinsic qualities and that the probability with which an object is preferred to another depends both on the difference between the qualities of the two competitors and on the reliability of the worker. We propose QUITE, a non-adaptive ranking algorithm that jointly estimates workers' reliabilities and qualities of objects. Performance of QUITE is compared in different scenarios against previously proposed algorithms. Finally, we show how QUITE can be naturally made adaptive.