Multi-object Classification via Crowdsourcing with a Reject Option

TL;DR

This paper proposes a weighted aggregation method for multi-object crowdsourcing classification tasks that incorporates a reject option for workers, improving accuracy by handling unreliable responses and greedy workers.

Contribution

It introduces a novel weighted voting scheme that accounts for worker reliability and reject options, along with adaptive strategies to mitigate greedy worker effects.

Findings

Weighted voting improves classification accuracy.

Reject options help filter unreliable responses.

Adaptive strategies effectively handle greedy workers.

Abstract

Consider designing an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final result. We consider the novel scenario where workers have a reject option so they may skip microtasks when they are unable or choose not to respond. For example, in mismatched speech transcription, workers who do not know the language may not be able to respond to microtasks focused on phonological dimensions outside their categorical perception. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize the crowd's classification performance. We evaluate system performance in both exact and asymptotic forms. Further, we consider the setting where there may be a set of greedy workers that complete microtasks even when they are unable to perform it reliably. We consider an oblivious and an expurgation strategy to deal with greedy workers, developing an algorithm to adaptively switch between the two based on the estimated fraction of greedy workers in the anonymous crowd. Simulation results show improved performance compared with conventional majority voting.