Evaluating the Crowd with Confidence

TL;DR

This paper introduces methods to generate confidence intervals for worker error rates in crowdsourcing, improving quality control by enabling better evaluation and management of worker performance.

Contribution

It presents novel techniques for calculating confidence intervals for worker error rates, applicable across various datasets and used for worker eviction and answer accuracy assessment.

Findings

Correct confidence intervals are generated on real-world datasets.

Techniques effectively identify poorly performing workers.

Confidence intervals improve evaluation of answer accuracy.

Abstract

Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers.