Optimal Inference in Crowdsourced Classification via Belief Propagation

TL;DR

This paper demonstrates that Belief Propagation achieves the fundamental limit of label inference accuracy in crowdsourced classification under the Dawid-Skene model, outperforming existing algorithms.

Contribution

It introduces a new lower bound on the fundamental limit and proves BP's optimality, bridging the gap between theory and practical performance.

Findings

BP matches the fundamental limit of inference accuracy.

BP outperforms state-of-the-art algorithms in experiments.

Theoretical analysis confirms BP's optimality across regimes.

Abstract

Crowdsourcing systems are popular for solving large-scale labelling tasks with low-paid workers. We study the problem of recovering the true labels from the possibly erroneous crowdsourced labels under the popular Dawid-Skene model. To address this inference problem, several algorithms have recently been proposed, but the best known guarantee is still significantly larger than the fundamental limit. We close this gap by introducing a tighter lower bound on the fundamental limit and proving that Belief Propagation (BP) exactly matches this lower bound. The guaranteed optimality of BP is the strongest in the sense that it is information-theoretically impossible for any other algorithm to correctly label a larger fraction of the tasks. Experimental results suggest that BP is close to optimal for all regimes considered and improves upon competing state-of-the-art algorithms.