Reliable Crowdsourcing for Multi-Class Labeling using Coding Theory

TL;DR

This paper introduces a coding-theory-based approach to improve the reliability of multi-class labeling in crowdsourcing systems, effectively handling unreliable workers and enhancing classification accuracy.

Contribution

It applies error-control coding and decoding algorithms to crowdsourcing, providing a novel framework for reliable multi-class labeling despite worker unreliability.

Findings

Coding-based schemes outperform majority voting.

Pairing workers and diversifying questions improve accuracy.

Effective across different crowd models.

Abstract

Crowdsourcing systems often have crowd workers that perform unreliable work on the task they are assigned. In this paper, we propose the use of error-control codes and decoding algorithms to design crowdsourcing systems for reliable classification despite unreliable crowd workers. Coding-theory based techniques also allow us to pose easy-to-answer binary questions to the crowd workers. We consider three different crowdsourcing models: systems with independent crowd workers, systems with peer-dependent reward schemes, and systems where workers have common sources of information. For each of these models, we analyze classification performance with the proposed coding-based scheme. We develop an ordering principle for the quality of crowds and describe how system performance changes with the quality of the crowd. We also show that pairing among workers and diversification of the questions help in improving system performance. We demonstrate the effectiveness of the proposed coding-based scheme using both simulated data and real datasets from Amazon Mechanical Turk, a crowdsourcing microtask platform. Results suggest that use of good codes may improve the performance of the crowdsourcing task over typical majority-voting approaches.