The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat

TL;DR
This paper explores the perspectivist paradigm shift in data labeling, emphasizing disagreement among annotators as a valuable source of information rather than a problem to be minimized.
Contribution
It critically examines traditional assumptions about disagreement in labeling and advocates for perspectivist approaches that leverage subjectivity for richer data insights.
Findings
Disagreement can be a valuable source of information.
Perspectivist approaches challenge traditional labeling assumptions.
Recommendations for improving data labeling practices.
Abstract
Longstanding data labeling practices in machine learning involve collecting and aggregating labels from multiple annotators. But what should we do when annotators disagree? Though annotator disagreement has long been seen as a problem to minimize, new perspectivist approaches challenge this assumption by treating disagreement as a valuable source of information. In this position paper, we examine practices and assumptions surrounding the causes of disagreement--some challenged by perspectivist approaches, and some that remain to be addressed--as well as practical and normative challenges for work operating under these assumptions. We conclude with recommendations for the data labeling pipeline and avenues for future research engaging with subjectivity and disagreement.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Ethics and Social Impacts of AI · Explainable Artificial Intelligence (XAI)
