A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality
Kenji Hata, Ranjay Krishna, Li Fei-Fei, Michael S. Bernstein

TL;DR
This study analyzes long-term worker quality in crowdsourcing, revealing stability over time and the ability to predict future performance from early tasks, challenging assumptions of fatigue or satisficing.
Contribution
It provides the first large-scale longitudinal analysis of crowd worker quality and demonstrates that early performance can reliably predict long-term quality.
Findings
Workers maintain consistent quality over extended periods.
Workers do not adjust quality based on acceptance thresholds.
Early task performance predicts long-term quality effectively.
Abstract
Microtask crowdsourcing is increasingly critical to the creation of extremely large datasets. As a result, crowd workers spend weeks or months repeating the exact same tasks, making it necessary to understand their behavior over these long periods of time. We utilize three large, longitudinal datasets of nine million annotations collected from Amazon Mechanical Turk to examine claims that workers fatigue or satisfice over these long periods, producing lower quality work. We find that, contrary to these claims, workers are extremely stable in their quality over the entire period. To understand whether workers set their quality based on the task's requirements for acceptance, we then perform an experiment where we vary the required quality for a large crowdsourcing task. Workers did not adjust their quality based on the acceptance threshold: workers who were above the threshold continued…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Evacuation and Crowd Dynamics · Human Mobility and Location-Based Analysis
