Data Quality in Crowdsourcing and Spamming Behavior Detection

TL;DR

This paper proposes a systematic approach to evaluate data quality and detect spamming in crowdsourcing, using variance decomposition, behavioral classification, and credibility metrics, demonstrated on face verification tasks.

Contribution

It introduces novel methods for assessing data quality and identifying spammers in crowdsourcing, including a spammer index and credibility metrics based on Markov chains and random effects models.

Findings

Effective detection of spammers with behavioral pattern classification.

Improved data quality assessment through variance decomposition.

Practical validation on real-world crowdsourcing data.

Abstract

As crowdsourcing emerges as an efficient and cost-effective method for obtaining labels for machine learning datasets, it is important to assess the quality of crowd-provided data, so as to improve analysis performance and reduce biases in subsequent machine learning tasks. Given the lack of ground truth in most cases of crowdsourcing, we refer to data quality as annotators' consistency and credibility. Unlike the simple scenarios where Kappa coefficient and intraclass correlation coefficient usually can apply, online crowdsourcing requires dealing with more complex situations. We introduce a systematic method for evaluating data quality and detecting spamming threats via variance decomposition, and we classify spammers into three categories based on their different behavioral patterns. A spammer index is proposed to assess entire data consistency, and two metrics are developed to measure crowd workers' credibility by utilizing the Markov chain and generalized random effects models. Furthermore, we showcase the practicality of our techniques and their advantages by applying them on a face verification task with both simulation and real-world data collected from two crowdsourcing platforms.