Towards Fair Pay and Equal Work: Imposing View Time Limits in Crowdsourced Image Classification
Gordon Lim, Stefan Larson, Yu Huang, Kevin Leach

TL;DR
This study investigates how imposing view time limits on crowdsourced image classification tasks affects worker performance, satisfaction, and data quality, suggesting that time limits can promote fairer pay and consistent effort.
Contribution
The paper provides empirical evidence that time limits can effectively balance data quality, worker satisfaction, and fair compensation in crowdsourcing tasks.
Findings
Performance impact decreases with longer view times
Consensus algorithms maintain data quality under time limits
Workers prefer shorter time limits
Abstract
Crowdsourcing is a common approach to rapidly annotate large volumes of data in machine learning applications. Typically, crowd workers are compensated with a flat rate based on an estimated completion time to meet a target hourly wage. Unfortunately, prior work has shown that variability in completion times among crowd workers led to overpayment by 168% in one case, and underpayment by 16% in another. However, by setting a time limit for task completion, it is possible to manage the risk of overpaying or underpaying while still facilitating flat rate payments. In this paper, we present an analysis of the impact of a time limit on crowd worker performance and satisfaction. We conducted a human study with a maximum view time for a crowdsourced image classification task. We find that the impact on overall crowd worker performance diminishes as view time increases. Despite some images…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAuction Theory and Applications · Mobile Crowdsensing and Crowdsourcing
