The Importance of Being Earnest in Crowdsourcing Systems

TL;DR

This paper investigates how utilizing worker reputation information can significantly enhance crowdsourcing system performance through formal optimization and heuristic algorithms, despite inaccuracies in reputation estimates.

Contribution

It formalizes the task assignment problem using reputation data, proposes a greedy heuristic algorithm, and evaluates the impact of reputation accuracy on system performance.

Findings

Reputation information can greatly improve task assignment efficiency.

Inaccurate reputation estimates still provide substantial benefits.

Combining task assignment with existing decision rules yields optimal results under high inaccuracy.

Abstract

This paper presents the first systematic investigation of the potential performance gains for crowdsourcing systems, deriving from available information at the requester about individual worker earnestness (reputation). In particular, we first formalize the optimal task assignment problem when workers' reputation estimates are available, as the maximization of a monotone (submodular) function subject to Matroid constraints. Then, being the optimal problem NP-hard, we propose a simple but efficient greedy heuristic task allocation algorithm. We also propose a simple ``maximum a-posteriori`` decision rule. Finally, we test and compare different solutions, showing that system performance can greatly benefit from information about workers' reputation. Our main findings are that: i) even largely inaccurate estimates of workers' reputation can be effectively exploited in the task assignment to greatly improve system performance; ii) the performance of the maximum a-posteriori decision rule quickly degrades as worker reputation estimates become inaccurate; iii) when workers' reputation estimates are significantly inaccurate, the best performance can be obtained by combining our proposed task assignment algorithm with the LRA decision rule introduced in the literature.