The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation

TL;DR

This paper critically examines the limitations of the traditional 'ground truth' paradigm in data annotation, highlighting systemic biases and proposing a shift towards embracing human disagreement as valuable sociotechnical signals.

Contribution

It offers a systematic review of recent research exposing systemic failures in annotation practices and proposes a roadmap for pluralistic, culturally aware annotation infrastructures.

Findings

Systemic failures in annotation practices reinforce Western norms.

Reliance on model-mediated annotations introduces bias and removes human voices.

Disagreement among annotators is a vital signal for cultural competence.

Abstract

In machine learning, "ground truth" refers to the assumed correct labels used to train and evaluate models. However, the foundational "ground truth" paradigm rests on a positivistic fallacy that treats human disagreement as technical noise rather than a vital sociotechnical signal. This systematic literature review analyzes research published between 2020 and 2025 across seven premier venues: ACL, AIES, CHI, CSCW, EAAMO, FAccT, and NeurIPS, investigating the mechanisms in data annotation practices that facilitate this "consensus trap". Our reflexive thematic analysis of 346 papers reveals that systemic failures in positional legibility, combined with the recent architectural shift toward human-as-verifier models, specifically the reliance on model-mediated annotations, introduce deep-seated anchoring bias and effectively remove human voices from the loop. We further demonstrate how geographic hegemony imposes Western norms as universal benchmarks, often enforced by the performative alignment of precarious data workers who prioritize requester compliance over honest subjectivity to avoid economic penalties. Critiquing the "noisy sensor" fallacy, where statistical models misdiagnose pluralism as error, we argue for reclaiming disagreement as a high-fidelity signal essential for building culturally competent models. To address these systemic tensions, we propose a roadmap for pluralistic annotation infrastructures that shift the objective from discovering a singular "right" answer to mapping the diversity of human experience.