Detecting adversaries in Crowdsourcing

TL;DR

This paper presents a method to detect and mitigate adversaries in crowdsourcing classification tasks by analyzing second-order moments of annotator responses, improving robustness against malicious participants.

Contribution

It introduces a novel approach leveraging second-order moments to identify and counteract adversaries in crowdsourcing, even when they cooperate or deviate arbitrarily.

Findings

Effective detection of adversaries demonstrated on synthetic datasets.

Robustness improvements shown on real crowdsourcing datasets.

Abstract

Despite its successes in various machine learning and data science tasks, crowdsourcing can be susceptible to attacks from dedicated adversaries. This work investigates the effects of adversaries on crowdsourced classification, under the popular Dawid and Skene model. The adversaries are allowed to deviate arbitrarily from the considered crowdsourcing model, and may potentially cooperate. To address this scenario, we develop an approach that leverages the structure of second-order moments of annotator responses, to identify large numbers of adversaries, and mitigate their impact on the crowdsourcing task. The potential of the proposed approach is empirically demonstrated on synthetic and real crowdsourcing datasets.