Training and Evaluating with Human Label Variation: An Empirical Study

TL;DR

This paper investigates human label variation in data annotation, proposing new differentiable evaluation metrics based on fuzzy set theory, and empirically compares various training methods and metrics across multiple datasets.

Contribution

It introduces novel differentiable evaluation metrics for human label variation and evaluates their effectiveness in training and assessing models.

Findings

Training on disaggregated annotations or soft labels yields the best performance.

Proposed soft micro F1 score is among the top metrics for HLV data.

Differentiable metrics as training objectives are less effective than label-based training.

Abstract

Human label variation (HLV) challenges the standard assumption that a labelled instance has a single ground truth, instead embracing the natural variation in human annotation to train and evaluate models. While various training methods and metrics for HLV have been proposed, it is still unclear which methods and metrics perform best in what settings. We propose new evaluation metrics for HLV leveraging fuzzy set theory. Since these new proposed metrics are differentiable, we then in turn experiment with employing these metrics as training objectives. We conduct an extensive study over 6 HLV datasets testing 14 training methods and 6 evaluation metrics. We find that training on either disaggregated annotations or soft labels performs best across metrics, outperforming training using the proposed training objectives with differentiable metrics. We also show that our proposed soft micro F1 score is one of the best metrics for HLV data.