Tuning Crowdsourced Human Computation

TL;DR

This paper models crowd workers as Human Processing Units and develops optimization algorithms to minimize job completion time within a fixed budget, considering different task complexities and validating results through simulations and experiments.

Contribution

It introduces a new framework for optimizing crowdsourcing performance by modeling workers as HPUs and devising budget allocation strategies for various task scenarios.

Findings

Optimal budget allocation reduces latency significantly.

Worker clock-rate depends on remuneration, affecting task completion times.

Validated strategies outperform baseline approaches in simulations and real experiments.

Abstract

As the use of crowdsourcing increases, it is important to think about performance optimization. For this purpose, it is possible to think about each worker as a HPU(Human Processing Unit), and to draw inspiration from performance optimization on traditional computers or cloud nodes with CPUs. However, as we characterize HPUs in detail for this purpose, we find that there are important differences between CPUs and HPUs, leading to the need for completely new optimization algorithms. In this paper, we study the specific optimization problem of obtaining results fastest for a crowd sourced job with a fixed total budget. In crowdsourcing, jobs are usually broken down into sets of small tasks, which are assigned to workers one at a time. We consider three scenarios of increasing complexity: Identical Round Homogeneous tasks, Multiplex Round Homogeneous tasks, and Multiple Round Heterogeneous tasks. For each scenario, we analyze the stochastic behavior of the HPU clock-rate as a function of the remuneration offered. After that, we develop an optimum Budget Allocation strategy to minimize the latency for job completion. We validate our results through extensive simulations and experiments on Amazon Mechanical Turk.