Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks

TL;DR

This paper introduces a schema-level diagnostic method for analyzing subjective NLP datasets, helping identify whether annotator disagreement stems from unclear criteria or overlapping categories, prior to final label assignment.

Contribution

It proposes a novel diagnostic approach that uses multi-annotator criterion judgments to audit annotation schemas before gold labels are finalized.

Findings

Disagreement mainly arises from unstable criteria and overlapping categories.

Nearly half of sentences activate multiple categories, indicating ambiguity.

The diagnostic aligns with expert disagreements, guiding schema refinement.

Abstract

Subjective NLP datasets typically aggregate annotator judgments into a single gold label, making it difficult to diagnose whether disagreement reflects unclear criteria, collapsed distinctions, or legitimate plurality. We propose a \emph{schema-level diagnostic} for auditing expert-designed annotation schemas \emph{prior to} gold-label commitment, using only multi-annotator criterion judgments. The diagnostic separates two failure modes: unstable criteria with hard-to-operationalize boundaries, and systematic overlap that blurs the boundaries between mutually exclusive categories. Applied to persuasive value extraction in commercial documents, we find that disagreement is not diffuse: instability concentrates in a few criteria, while nearly half of covered sentences activate multiple categories. These signals align with where domain experts disagree, yielding an evidence-based audit for tightening guidelines, revising category structure, or reconsidering the annotation paradigm.