Multicategory Crowdsourcing Accounting for Plurality in Worker Skill and Intention, Task Difficulty, and Task Heterogeneity

TL;DR

This paper introduces new stochastic and deterministic methods for aggregating multicategory crowdsourced data, explicitly modeling worker skill, intention, and task difficulty to improve accuracy in diverse and challenging crowdsourcing scenarios.

Contribution

It proposes novel crowd aggregation techniques that account for worker heterogeneity and task complexity, outperforming traditional methods especially with unskilled or malicious workers.

Findings

Methods outperform traditional aggregation in heterogeneous settings

Explicit modeling improves accuracy with minority skilled workers

Approaches are effective in both supervised and unsupervised contexts

Abstract

Crowdsourcing allows to instantly recruit workers on the web to annotate image, web page, or document databases. However, worker unreliability prevents taking a workers responses at face value. Thus, responses from multiple workers are typically aggregated to more reliably infer ground-truth answers. We study two approaches for crowd aggregation on multicategory answer spaces stochastic modeling based and deterministic objective function based. Our stochastic model for answer generation plausibly captures the interplay between worker skills, intentions, and task difficulties and allows us to model a broad range of worker types. Our deterministic objective based approach does not assume a model for worker response generation. Instead, it aims to maximize the average aggregate confidence of weighted plurality crowd decision making. In both approaches, we explicitly model the skill and intention of individual workers, which is exploited for improved crowd aggregation. Our methods are applicable in both unsupervised and semisupervised settings, and also when the batch of tasks is heterogeneous. As observed experimentally, the proposed methods can defeat tyranny of the masses, they are especially advantageous when there is a minority of skilled workers amongst a large crowd of unskilled and malicious workers.