Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation

TL;DR

This paper reviews various inter-annotator agreement metrics in NLP, discussing their assumptions, limitations, and best practices for reliable annotation and evaluation.

Contribution

It provides a comprehensive guide for selecting and interpreting IAA metrics tailored to different NLP annotation tasks.

Findings

Organizes agreement measures by task type

Highlights impact of label imbalance and missing data

Recommends best practices for reporting agreement

Abstract

Human annotation remains the foundation of reliable and interpretable data in Natural Language Processing (NLP). As annotation and evaluation tasks continue to expand, from categorical labelling to segmentation, subjective judgment, and continuous rating, measuring agreement between annotators has become increasingly more complex. This paper outlines how inter-annotator agreement (IAA) has been conceptualised and applied across NLP and related disciplines, describing the assumptions and limitations of common approaches. We organise agreement measures by task type and discuss how factors such as label imbalance and missing data influence reliability estimates. In addition, we highlight best practices for clear and transparent reporting, including the use of confidence intervals and the analysis of disagreement patterns. The paper aims to serve as a guide for selecting and interpreting agreement measures, promoting more consistent and reproducible human annotation and evaluation in NLP.