Supervised Collective Classification for Crowdsourcing

TL;DR

This paper introduces a supervised algorithm for collective classification in crowdsourcing, leveraging known labels to identify reliable labelers and improve overall accuracy over traditional unsupervised methods.

Contribution

It presents a novel supervised approach that estimates labeler reliability using a saddle point algorithm, outperforming existing unsupervised algorithms.

Findings

Supervised methods achieve higher classification accuracy.

The proposed algorithm outperforms existing algorithms.

Reliability estimation improves label aggregation quality.

Abstract

Crowdsourcing utilizes the wisdom of crowds for collective classification via information (e.g., labels of an item) provided by labelers. Current crowdsourcing algorithms are mainly unsupervised methods that are unaware of the quality of crowdsourced data. In this paper, we propose a supervised collective classification algorithm that aims to identify reliable labelers from the training data (e.g., items with known labels). The reliability (i.e., weighting factor) of each labeler is determined via a saddle point algorithm. The results on several crowdsourced data show that supervised methods can achieve better classification accuracy than unsupervised methods, and our proposed method outperforms other algorithms.