Crowd Access Path Optimization: Diversity Matters

TL;DR

This paper introduces the Access Path Model (APM), a novel approach for optimizing crowd access by considering diversity and correlation among workers, leading to cost-effective quality assurance in crowdsourcing.

Contribution

The paper proposes the APM, a new crowd model that leverages access paths and a greedy algorithm for optimized, cost-efficient crowdsourcing with high quality.

Findings

APM improves cost-efficiency in crowdsourcing tasks.

The greedy algorithm provides a provably good approximate plan.

Experimental results on three datasets demonstrate effectiveness in large-scale scenarios.

Abstract

Quality assurance is one the most important challenges in crowdsourcing. Assigning tasks to several workers to increase quality through redundant answers can be expensive if asking homogeneous sources. This limitation has been overlooked by current crowdsourcing platforms resulting therefore in costly solutions. In order to achieve desirable cost-quality tradeoffs it is essential to apply efficient crowd access optimization techniques. Our work argues that optimization needs to be aware of diversity and correlation of information within groups of individuals so that crowdsourcing redundancy can be adequately planned beforehand. Based on this intuitive idea, we introduce the Access Path Model (APM), a novel crowd model that leverages the notion of access paths as an alternative way of retrieving information. APM aggregates answers ensuring high quality and meaningful confidence. Moreover, we devise a greedy optimization algorithm for this model that finds a provably good approximate plan to access the crowd. We evaluate our approach on three crowdsourced datasets that illustrate various aspects of the problem. Our results show that the Access Path Model combined with greedy optimization is cost-efficient and practical to overcome common difficulties in large-scale crowdsourcing like data sparsity and anonymity.