A General Early-Stopping Module for Crowdsourced Ranking

TL;DR

This paper introduces a statistical early-stopping module for crowdsourced ranking tasks that can be integrated with existing algorithms to save budget while maintaining high ranking quality.

Contribution

We propose a novel early-stopping module based on statistical quality estimation, improving budget efficiency in crowdsourced ranking without sacrificing accuracy.

Findings

Our module outperforms existing stopping criteria in experiments.

It achieves high-quality rankings with fewer microtasks.

The system is practical and effective in real-world scenarios.

Abstract

Crowdsourcing can be used to determine a total order for an object set (e.g., the top-10 NBA players) based on crowd opinions. This ranking problem is often decomposed into a set of microtasks (e.g., pairwise comparisons). These microtasks are passed to a large number of workers and their answers are aggregated to infer the ranking. The number of microtasks depends on the budget allocated for the problem. Intuitively, the higher the number of microtask answers, the more accurate the ranking becomes. However, it is often hard to decide the budget required for an accurate ranking. We study how a ranking process can be terminated early, and yet achieve a high-quality ranking and great savings in the budget. We use statistical tools to estimate the quality of the ranking result at any stage of the crowdsourcing process and terminate the process as soon as the desired quality is achieved. Our proposed early-stopping module can be seamlessly integrated with most existing inference algorithms and task assignment methods. We conduct extensive experiments and show that our early-stopping module is better than other existing general stopping criteria. We also implement a prototype system to demonstrate the usability and effectiveness of our approach in practice.