Human Label Variation in Implicit Discourse Relation Recognition

TL;DR

This paper investigates how human judgment variability affects Implicit Discourse Relation Recognition, showing that models trained on label distributions are more stable than annotator-specific models, especially in cognitively demanding cases.

Contribution

It compares distribution-based and perspectivist models in IDRR, highlighting the limitations of perspectivist approaches in highly ambiguous, cognitively complex cases.

Findings

Models trained on label distributions are more stable.

Annotator-specific models perform poorly without ambiguity reduction.

Cognitively demanding cases cause high human disagreement.

Abstract

There is growing recognition that many NLP tasks lack a single ground truth, as human judgments reflect diverse perspectives. To capture this variation, models have been developed to predict full annotation distributions rather than majority labels, while perspectivist models aim to reproduce the interpretations of individual annotators. In this work, we compare these approaches on Implicit Discourse Relation Recognition (IDRR), a highly ambiguous task where disagreement often arises from cognitive complexity rather than ideological bias. Our experiments show that existing annotator-specific models perform poorly in IDRR unless ambiguity is reduced, whereas models trained on label distributions yield more stable predictions. Further analysis indicates that frequent cognitively demanding cases drive inconsistency in human interpretation, posing challenges for perspectivist modeling in IDRR.