A Streaming Algorithm for Crowdsourced Data Classification

TL;DR

This paper introduces a streaming algorithm for binary data classification in crowdsourcing that learns labeller accuracy over time, providing optimal performance guarantees and outperforming existing methods in experiments.

Contribution

The paper presents a novel streaming algorithm that adaptively learns labeller competence and achieves finite regret compared to the optimal classifier with known error rates.

Findings

Algorithm is optimal with finite cumulative regret.

Complexity is linear in number of labellers and tasks.

Performs better than majority voting and EM in experiments.

Abstract

We propose a streaming algorithm for the binary classification of data based on crowdsourcing. The algorithm learns the competence of each labeller by comparing her labels to those of other labellers on the same tasks and uses this information to minimize the prediction error rate on each task. We provide performance guarantees of our algorithm for a fixed population of independent labellers. In particular, we show that our algorithm is optimal in the sense that the cumulative regret compared to the optimal decision with known labeller error probabilities is finite, independently of the number of tasks to label. The complexity of the algorithm is linear in the number of labellers and the number of tasks, up to some logarithmic factors. Numerical experiments illustrate the performance of our algorithm compared to existing algorithms, including simple majority voting and expectation-maximization algorithms, on both synthetic and real datasets.