Quantifying and Predicting Disagreement in Graded Human Ratings

TL;DR

This paper explores the patterns of disagreement among human annotators on offensive language detection, proposing a new metric to quantify opposing perspectives and predicting annotation variance from textual features.

Contribution

It introduces the Opposition Index to measure perspective opposition and demonstrates methods to predict annotation disagreement from textual data.

Findings

Moderate correlation between estimated and observed annotation variance.

Two approaches perform similarly in variance prediction: direct prediction and distribution estimation.

High opposition index items are harder to predict and often underestimated by models.

Abstract

It is increasingly recognized that human annotators do not always agree, and such disagreement is inherent in many annotation tasks. However, not all instances in a given task elicit the same degree of opinion divergence. In this paper, we investigate annotation variation patterns in graded human ratings for inappropriate languages, including offensive language, hate speech, and toxic language perception. We examine whether the degree of annotation disagreement can be predicted from textual features. We further propose the Opposition Index, a metric that quantifies perspective opposition among annotators on a given item, and investigate the predictability of instances with potentially opposing human opinions. Our results show a moderate positive correlation between estimated and observed annotation variance. We find that two approaches achieve comparable performance in variance prediction: directly predicting the variance value and estimating it from predicted annotation distributions. Our results on opposition perspective prediction show that items with high opposition index values are more difficult to predict and are often underestimated by models.