Optimal Crowdsourced Classification with a Reject Option in the Presence of Spammers

TL;DR

This paper proposes an optimized weighted majority voting scheme for crowdsourced $M$-ary classification tasks that incorporates a reject option for workers, aiming to improve accuracy despite potential spammers.

Contribution

It introduces a novel weighted aggregation method that accounts for worker reject options, enhancing classification performance in crowdsourcing systems.

Findings

Optimized weights improve classification accuracy.

Reject options help mitigate spam influence.

Method outperforms traditional voting schemes.

Abstract

We explore the design of an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance.