TrueLabel + Confusions: A Spectrum of Probabilistic Models in Analyzing Multiple Ratings

TL;DR

This paper introduces a hierarchical Bayesian model called HybridConfusion that extends the DawidSkene model to analyze multiple ratings, providing diagnostic insights into judges and outperforming previous models on various datasets.

Contribution

The paper generalizes the DawidSkene model into a spectrum of probabilistic models under the 'TrueLabel + Confusion' paradigm, with the HybridConfusion model showing improved performance.

Findings

HybridConfusion outperforms DawidSkene on synthetic data.

HybridConfusion provides better diagnostic insights into judges.

The model is effective on real-world datasets.

Abstract

This paper revisits the problem of analyzing multiple ratings given by different judges. Different from previous work that focuses on distilling the true labels from noisy crowdsourcing ratings, we emphasize gaining diagnostic insights into our in-house well-trained judges. We generalize the well-known DawidSkene model (Dawid & Skene, 1979) to a spectrum of probabilistic models under the same "TrueLabel + Confusion" paradigm, and show that our proposed hierarchical Bayesian model, called HybridConfusion, consistently outperforms DawidSkene on both synthetic and real-world data sets.