A Study of Unsupervised Adaptive Crowdsourcing

TL;DR

This paper investigates unsupervised crowdsourcing performance by analyzing response correlation with majority responses, considering both multiple independent tasks and a single large task, highlighting the impact of crowd reliability.

Contribution

It introduces a model for unsupervised crowdsourcing that assesses performance based on response correlation, providing insights into reliability effects in different task settings.

Findings

Response correlation with majority indicates crowd reliability.

Performance varies with independent vs. large single assignment.

Reliability significantly influences crowdsourcing effectiveness.

Abstract

We consider unsupervised crowdsourcing performance based on the model wherein the responses of end-users are essentially rated according to how their responses correlate with the majority of other responses to the same subtasks/questions. In one setting, we consider an independent sequence of identically distributed crowdsourcing assignments (meta-tasks), while in the other we consider a single assignment with a large number of component subtasks. Both problems yield intuitive results in which the overall reliability of the crowd is a factor.