Dense Limit of the Dawid-Skene Model for Crowdsourcing and Regions of Sub-optimality of Message Passing Algorithms

TL;DR

This paper analyzes the dense limit of the Dawid-Skene model in crowdsourcing, providing a closed-form Bayes-optimal performance, identifying regions where message passing algorithms are optimal or sub-optimal, and testing these findings on real data.

Contribution

It extends the understanding of the Dawid-Skene model by connecting it to low-rank matrix estimation and characterizing the optimality regions of message passing algorithms.

Findings

Bayes-optimal performance expressed in closed form.

Identification of parameter regions where message passing is optimal.

Numerical validation on sparse instances and real datasets.

Abstract

Crowdsourcing is a strategy to categorize data through the contribution of many individuals. A wide range of theoretical and algorithmic contributions are based on the model of Dawid and Skene [1]. Recently it was shown in [2,3] that, in certain regimes, belief propagation is asymptotically optimal for data generated from the Dawid-Skene model. This paper is motivated by this recent progress. We analyze the dense limit of the Dawid-Skene model. It is shown that it belongs to a larger class of low-rank matrix estimation problems for which it is possible to express the asymptotic, Bayes-optimal, performance in a simple closed form. In the dense limit the mapping to a low-rank matrix estimation problem provides an approximate message passing algorithm that solves the problem algorithmically. We identify the regions where the algorithm efficiently computes the Bayes-optimal estimates. Our analysis refines the results of [2,3] about optimality of message passing algorithms by characterizing regions of parameters where these algorithms do not match the Bayes-optimal performance. We further study numerically the performance of approximate message passing, derived in the dense limit, on sparse instances and carry out experiments on a real world dataset.