Label Selection Approach to Learning from Crowds

TL;DR

This paper introduces a novel label selection layer for learning from crowdsourced noisy labels, improving model training by automatically selecting reliable worker labels without assuming specific noise models.

Contribution

It proposes a versatile label selection approach that can be integrated into various supervised learning models, enhancing robustness to label noise from crowdsourcing.

Findings

Performance is comparable or superior to Crowd Layer in classification tasks.

Method is applicable to almost all supervised learning variants.

Outperforms existing methods in most cases, except regression.

Abstract

Supervised learning, especially supervised deep learning, requires large amounts of labeled data. One approach to collect large amounts of labeled data is by using a crowdsourcing platform where numerous workers perform the annotation tasks. However, the annotation results often contain label noise, as the annotation skills vary depending on the crowd workers and their ability to complete the task correctly. Learning from Crowds is a framework which directly trains the models using noisy labeled data from crowd workers. In this study, we propose a novel Learning from Crowds model, inspired by SelectiveNet proposed for the selective prediction problem. The proposed method called Label Selection Layer trains a prediction model by automatically determining whether to use a worker's label for training using a selector network. A major advantage of the proposed method is that it can be applied to almost all variants of supervised learning problems by simply adding a selector network and changing the objective function for existing models, without explicitly assuming a model of the noise in crowd annotations. The experimental results show that the performance of the proposed method is almost equivalent to or better than the Crowd Layer, which is one of the state-of-the-art methods for Deep Learning from Crowds, except for the regression problem case.