Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP

TL;DR

This survey reviews recent methods in NLP that explicitly model annotator disagreement, emphasizing a shift from consensus to perspective-aware approaches to better capture subjective and ambiguous task nuances.

Contribution

It provides a unified taxonomy of disagreement sources, synthesizes modeling frameworks, and discusses evaluation metrics and future challenges in disagreement-aware NLP.

Findings

Modeling disagreement enhances understanding of subjective tasks.

Shift from consensus to perspectivist modeling improves interpretability.

Evaluation metrics for disagreement are mostly descriptive, not normative.

Abstract

Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagreement as noise to be removed, recent work increasingly models it as a meaningful signal reflecting variation in interpretation and perspective. This survey provides a unified view of disagreement-aware NLP methods. We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors. We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure, highlighting a shift from consensus learning toward explicitly modeling disagreement, and toward capturing structured relationships among annotators. We review evaluation metrics for both predictive performance and annotator behavior, and noting that most fairness evaluations remain descriptive rather than normative. We conclude by identifying open challenges and future directions, including integrating multiple sources of variation, developing disagreement-aware interpretability frameworks, and grappling with the practical tradeoffs of perspectivist modeling.