Improving Label Quality by Jointly Modeling Items and Annotators

TL;DR

This paper introduces a Bayesian framework that jointly models items and annotators to improve label quality from noisy data, enhancing ground truth estimation for supervised learning.

Contribution

It presents a scalable Bayesian soft clustering model that fully incorporates label distributions and combines graphical and neural models for better label quality estimation.

Findings

Improved ground truth estimates from noisy annotations.

Enhanced supervised learning performance using the proposed models.

Outperforms baseline and state-of-the-art models in experiments.

Abstract

We propose a fully Bayesian framework for learning ground truth labels from noisy annotators. Our framework ensures scalability by factoring a generative, Bayesian soft clustering model over label distributions into the classic David and Skene joint annotator-data model. Earlier research along these lines has neither fully incorporated label distributions nor explored clustering by annotators only or data only. Our framework incorporates all of these properties as: (1) a graphical model designed to provide better ground truth estimates of annotator responses as input to \emph{any} black box supervised learning algorithm, and (2) a standalone neural model whose internal structure captures many of the properties of the graphical model. We conduct supervised learning experiments using both models and compare them to the performance of one baseline and a state-of-the-art model.