Validating Label Consistency in NER Data Annotation

TL;DR

This paper introduces an empirical method to detect label inconsistencies in NER datasets, which helps improve annotation quality and model performance validation.

Contribution

It presents a novel approach to identify label inconsistencies across multiple NER datasets, validated on SCIERC and CoNLL03 datasets.

Findings

Detected 26.7% label mistakes in SCIERC test data

Identified 5.4% label mistakes in CoNLL03 test data

Validated label consistency after correction

Abstract

Data annotation plays a crucial role in ensuring your named entity recognition (NER) projects are trained with the right information to learn from. Producing the most accurate labels is a challenge due to the complexity involved with annotation. Label inconsistency between multiple subsets of data annotation (e.g., training set and test set, or multiple training subsets) is an indicator of label mistakes. In this work, we present an empirical method to explore the relationship between label (in-)consistency and NER model performance. It can be used to validate the label consistency (or catches the inconsistency) in multiple sets of NER data annotation. In experiments, our method identified the label inconsistency of test data in SCIERC and CoNLL03 datasets (with 26.7% and 5.4% label mistakes). It validated the consistency in the corrected version of both datasets.