Crowdsourcing with Difficulty: A Bayesian Rating Model for Heterogeneous Items

TL;DR

This paper introduces a Bayesian rating model that accounts for item difficulty and other heterogeneity factors, improving consensus inference in crowdsourced data with noisy and biased labels.

Contribution

The study develops a new measurement-error model that captures item-level effects and constrains posterior bimodality, enhancing the accuracy of crowdsourcing data analysis.

Findings

The new model fits data better than Dawid and Skene's model.

It effectively accounts for item heterogeneity in crowdsourced ratings.

Posterior predictive checks validate the model's goodness of fit.

Abstract

In applied statistics and machine learning, the "gold standards" used for training are often biased and almost always noisy. Dawid and Skene's justifiably popular crowdsourcing model adjusts for rater (coder, annotator) sensitivity and specificity, but fails to capture distributional properties of rating data gathered for training, which in turn biases training. In this study, we introduce a general purpose measurement-error model with which we can infer consensus categories by adding item-level effects for difficulty, discriminativeness, and guessability. We further show how to constrain the bimodal posterior of these models to avoid (or if necessary, allow) adversarial raters. We validate our model's goodness of fit with posterior predictive checks, the Bayesian analogue of $\chi^2$ tests. Dawid and Skene's model is rejected by goodness of fit tests, whereas our new model, which adjusts for item heterogeneity, is not rejected. We illustrate our new model with two well-studied data sets, binary rating data for caries in dental X-rays and implication in natural language.