Towards Fair Pay and Equal Work: Imposing View Time Limits in Crowdsourced Image Classification

TL;DR

This study investigates how imposing view time limits on crowdsourced image classification tasks affects worker performance, satisfaction, and data quality, suggesting that time limits can promote fairer pay and consistent effort.

Contribution

The paper provides empirical evidence that time limits can effectively balance data quality, worker satisfaction, and fair compensation in crowdsourcing tasks.

Findings

Performance impact decreases with longer view times

Consensus algorithms maintain data quality under time limits

Workers prefer shorter time limits

Abstract

Crowdsourcing is a common approach to rapidly annotate large volumes of data in machine learning applications. Typically, crowd workers are compensated with a flat rate based on an estimated completion time to meet a target hourly wage. Unfortunately, prior work has shown that variability in completion times among crowd workers led to overpayment by 168% in one case, and underpayment by 16% in another. However, by setting a time limit for task completion, it is possible to manage the risk of overpaying or underpaying while still facilitating flat rate payments. In this paper, we present an analysis of the impact of a time limit on crowd worker performance and satisfaction. We conducted a human study with a maximum view time for a crowdsourced image classification task. We find that the impact on overall crowd worker performance diminishes as view time increases. Despite some images being challenging under time limits, a consensus algorithm remains effective at preserving data quality and filters images needing more time. Additionally, crowd workers' consistent performance throughout the time-limited task indicates sustained effort, and their psychometric questionnaire scores show they prefer shorter limits. Based on our findings, we recommend implementing task time limits as a practical approach to making compensation more equitable and predictable.