End-to-End Learning from Noisy Crowd to Supervised Machine Learning Models

TL;DR

This paper proposes an end-to-end hybrid approach combining deep models and human experts to improve label quality from noisy crowd-sourced data, significantly enhancing supervised learning accuracy.

Contribution

It introduces an online label aggregation framework that estimates annotator confusion matrices and uses expert relabeling to reduce noise and improve classification performance.

Findings

Label error rate reduced by over 30% with aggregation.

Relabeling 10% of data yields over 90% accuracy.

Effective on image datasets like UCI and CIFAR-10.

Abstract

Labeling real-world datasets is time consuming but indispensable for supervised machine learning models. A common solution is to distribute the labeling task across a large number of non-expert workers via crowd-sourcing. Due to the varying background and experience of crowd workers, the obtained labels are highly prone to errors and even detrimental to the learning models. In this paper, we advocate using hybrid intelligence, i.e., combining deep models and human experts, to design an end-to-end learning framework from noisy crowd-sourced data, especially in an on-line scenario. We first summarize the state-of-the-art solutions that address the challenges of noisy labels from non-expert crowd and learn from multiple annotators. We show how label aggregation can benefit from estimating the annotators' confusion matrices to improve the learning process. Moreover, with the help of an expert labeler as well as classifiers, we cleanse aggregated labels of highly informative samples to enhance the final classification accuracy. We demonstrate the effectiveness of our strategies on several image datasets, i.e. UCI and CIFAR-10, using SVM and deep neural networks. Our evaluation shows that our on-line label aggregation with confusion matrix estimation reduces the error rate of labels by over 30%. Furthermore, relabeling only 10% of the data using the expert's results in over 90% classification accuracy with SVM.