Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing

TL;DR

This paper examines the unfair qualification requirements in crowdsourcing, revealing that workers spend extensive unpaid effort to qualify for better paid tasks, and proposes alternatives to improve fairness without compromising data quality.

Contribution

It analyzes the impact of qualification requirements on workers and introduces methods to reduce unpaid effort while maintaining high data quality in crowdsourcing.

Findings

Workers spend about 2.25 months of full-time effort to qualify for better paid tasks.

Alternative qualification methods can reduce unpaid effort without sacrificing data quality.

High-quality data can be collected with less burdensome qualification processes.

Abstract

Extensive work has argued in favour of paying crowd workers a wage that is at least equivalent to the U.S. federal minimum wage. Meanwhile, research on collecting high quality annotations suggests using a qualification that requires workers to have previously completed a certain number of tasks. If most requesters who pay fairly require workers to have completed a large number of tasks already then workers need to complete a substantial amount of poorly paid work before they can earn a fair wage. Through analysis of worker discussions and guidance for researchers, we estimate that workers spend approximately 2.25 months of full time effort on poorly paid tasks in order to get the qualifications needed for better paid tasks. We discuss alternatives to this qualification and conduct a study of the correlation between qualifications and work quality on two NLP tasks. We find that it is possible to reduce the burden on workers while still collecting high quality data.