Eliciting and Learning with Soft Labels from Every Annotator

TL;DR

This paper introduces a method for efficiently collecting soft labels from individual annotators to improve machine learning model performance, reducing the need for multiple annotators and providing a new dataset for research.

Contribution

The paper presents a novel elicitation methodology for soft labels, a new dataset CIFAR-10S, and demonstrates comparable model performance with fewer annotators.

Findings

Learning with soft labels improves model performance.

Fewer annotators are needed to achieve similar results.

The elicitation process has significant temporal costs.

Abstract

The labels used to train machine learning (ML) models are of paramount importance. Typically for ML classification tasks, datasets contain hard labels, yet learning using soft labels has been shown to yield benefits for model generalization, robustness, and calibration. Earlier work found success in forming soft labels from multiple annotators' hard labels; however, this approach may not converge to the best labels and necessitates many annotators, which can be expensive and inefficient. We focus on efficiently eliciting soft labels from individual annotators. We collect and release a dataset of soft labels (which we call CIFAR-10S) over the CIFAR-10 test set via a crowdsourcing study (N=248). We demonstrate that learning with our labels achieves comparable model performance to prior approaches while requiring far fewer annotators -- albeit with significant temporal costs per elicitation. Our elicitation methodology therefore shows nuanced promise in enabling practitioners to enjoy the benefits of improved model performance and reliability with fewer annotators, and serves as a guide for future dataset curators on the benefits of leveraging richer information, such as categorical uncertainty, from individual annotators.