Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing

TL;DR

This study evaluates the effectiveness of using the same qualification test across multiple task types in crowdsourcing, revealing that test difficulty influences worker accuracy and that Masters Qualification does not always guarantee better performance.

Contribution

It investigates the performance of inter-microtask qualification tests across different task types, highlighting their impact on worker accuracy and the limitations of Masters Qualification.

Findings

Difficult qualification tests lead to higher accuracy regardless of task difficulty.

Masters Qualification improves accuracy on low-difficulty tasks but not on high-difficulty tasks.

Qualification test difficulty affects worker filtering effectiveness.

Abstract

Qualification tests in crowdsourcing are often used to pre-filter workers by measuring their ability in executing microtasks.While creating qualification tests for each task type is considered as a common and reasonable way, this study investigates into its worker-filtering performance when the same qualification test is used across multiple types of tasks.On Amazon Mechanical Turk, we tested the annotation accuracy in six different cases where tasks consisted of two different difficulty levels, arising from the identical real-world domain: four combinatory cases in which the qualification test and the actual task were the same or different from each other, as well as two other cases where workers with Masters Qualification were asked to perform the actual task only.The experimental results demonstrated the two following findings: i) Workers that were assigned to a difficult qualification test scored better annotation accuracy regardless of the difficulty of the actual task; ii) Workers with Masters Qualification scored better annotation accuracy on the low-difficulty task, but were not as accurate as those who passed a qualification test on the high-difficulty task.