Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?

TL;DR

This paper models crowdsourcing tasks and workers as an examination process, proposing a probabilistic approach to score worker ability and task difficulty to improve the accuracy of aggregated results.

Contribution

It introduces a novel probabilistic model that classifies workers and tasks, and iteratively infers ground truth, worker ability, and task difficulty in crowdsourcing.

Findings

Effective in simulated environments

Validated on real crowdsourcing data

Improves task scoring accuracy

Abstract

Crowdsourcing is an online outsourcing mode which can solve the current machine learning algorithm's urge need for massive labeled data. Requester posts tasks on crowdsourcing platforms, which employ online workers over the Internet to complete tasks, then aggregate and return results to requester. How to model the interaction between different types of workers and tasks is a hot spot. In this paper, we try to model workers as four types based on their ability: expert, normal worker, sloppy worker and spammer, and divide tasks into hard, medium and easy task according to their difficulty. We believe that even experts struggle with difficult tasks while sloppy workers can get easy tasks right, and spammers always give out wrong answers deliberately. So, good examination tasks should have moderate degree of difficulty and discriminability to score workers more objectively. Thus, we first score workers' ability mainly on the medium difficult tasks, then reducing the weight of answers from sloppy workers and modifying the answers from spammers when inferring the tasks' ground truth. A probability graph model is adopted to simulate the task execution process, and an iterative method is adopted to calculate and update the ground truth, the ability of workers and the difficulty of the task successively. We verify the rightness and effectiveness of our algorithm both in simulated and real crowdsourcing scenes.