Embracing Error to Enable Rapid Crowdsourcing
Ranjay Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A., Shamma, Li Fei-Fei, Michael S. Bernstein

TL;DR
This paper introduces a rapid crowdsourcing method that accepts errors to significantly increase labeling speed, using response latency and task randomization to maintain accuracy across various tasks.
Contribution
It presents a novel approach that allows for faster crowdsourcing by embracing errors and correcting them through response modeling and task randomization.
Findings
Achieves up to 10x speedup over traditional methods
Effective across diverse labeling tasks
Errors can be rectified without sacrificing accuracy
Abstract
Microtask crowdsourcing has enabled dataset advances in social science and machine learning, but existing crowdsourcing schemes are too expensive to scale up with the expanding volume of data. To scale and widen the applicability of crowdsourcing, we present a technique that produces extremely rapid judgments for binary and categorical labels. Rather than punishing all errors, which causes workers to proceed slowly and deliberately, our technique speeds up workers' judgments to the point where errors are acceptable and even expected. We demonstrate that it is possible to rectify these errors by randomizing task order and modeling response latency. We evaluate our technique on a breadth of common labeling tasks such as image verification, word similarity, sentiment analysis and topic classification. Where prior work typically achieves a 0.25x to 1x speedup over fixed majority vote, our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
