Robust Deep Learning from Crowds with Belief Propagation

TL;DR

This paper introduces a neural-powered Bayesian framework for crowdsourcing that leverages belief propagation to improve robustness against noisy labels and overfitting, outperforming existing methods.

Contribution

It proposes deepBP, a novel belief propagation-based approach, and unifies existing methods under a Bayesian neural network framework for crowdsourcing.

Findings

deepBP is more robust against wrong priors

deepBP handles feature overfitting better

deepBP is resilient to extreme worker noise

Abstract

Crowdsourcing systems enable us to collect large-scale dataset, but inherently suffer from noisy labels of low-paid workers. We address the inference and learning problems using such a crowdsourced dataset with noise. Due to the nature of sparsity in crowdsourcing, it is critical to exploit both probabilistic model to capture worker prior and neural network to extract task feature despite risks from wrong prior and overfitted feature in practice. We hence establish a neural-powered Bayesian framework, from which we devise deepMF and deepBP with different choice of variational approximation methods, mean field (MF) and belief propagation (BP), respectively. This provides a unified view of existing methods, which are special cases of deepMF with different priors. In addition, our empirical study suggests that deepBP is a new approach, which is more robust against wrong prior, feature overfitting and extreme workers thanks to the more sophisticated BP than MF.