Same Target, Different Basins: Hard vs. Soft Labels for Annotator Distributions

TL;DR

This paper investigates hard-label methods as an alternative to soft-label training for annotator disagreement, demonstrating their effectiveness especially with sparse annotations and their impact on model convergence and robustness.

Contribution

It introduces and evaluates multipass and stochastic label sampling (SLS) methods for handling annotator disagreement, showing their advantages over traditional soft-label training under various conditions.

Findings

Hard-label methods outperform soft-label training with sparse annotations.

Both hard-label methods match soft-label performance when full annotator distributions are available.

Hard-label delivery leads to flatter model basins and improved out-of-distribution detection.

Abstract

When annotators disagree, that disagreement can reflect epistemic uncertainty rather than simple label noise. We study hard-label delivery as an alternative to the usual choices of collapsing votes to a single label or training directly on the empirical soft-label distribution. We focus on two primary hard-label methods: multipass, which cycles through observed votes while keeping the dataset size fixed, and stochastic label sampling (SLS), which samples one label per example at the start of each epoch. On CIFAR-10H, we find that when only a small number of annotations per example is available, hard-label delivery improves over soft-label training, with larger improvements where the sparse empirical target is farther from the full annotator distribution. When full annotator distributions are available, both hard-label methods match soft-label training. We use deterministic control as an ablation of multipass and shuffled SLS as a control that breaks the example-to-distribution match. We also show that SLS and soft-label cross-entropy optimize the same expected objective. Hard-label delivery also converges to flatter basins, with supporting descriptive evidence from OOD detection on SVHN and CIFAR-100. Overall, these results suggest that multipass is a strong practical default when raw vote counts are available, while SLS offers a lightweight alternative that remains competitive when only a few votes per example are available and matches soft-label training when full annotator distributions are available.