Named Entity Recognition -- Is there a glass ceiling?

TL;DR

This paper analyzes the persistent errors in state-of-the-art NER models, revealing limitations and proposing new techniques for improvement based on detailed error diagnostics on the CoNLL 2003 dataset.

Contribution

It provides a comprehensive error analysis of leading NER models and introduces new methods for annotation, training, and quality assessment.

Findings

Identifies specific error types that remain challenging for current models.

Highlights shared limitations across different NER architectures.

Proposes new diagnostic datasets and annotation techniques for better model evaluation.

Abstract

Recent developments in Named Entity Recognition (NER) have resulted in better and better models. However, is there a glass ceiling? Do we know which types of errors are still hard or even impossible to correct? In this paper, we present a detailed analysis of the types of errors in state-of-the-art machine learning (ML) methods. Our study reveals the weak and strong points of the Stanford, CMU, FLAIR, ELMO and BERT models, as well as their shared limitations. We also introduce new techniques for improving annotation, for training processes and for checking a model's quality and stability. Presented results are based on the CoNLL 2003 data set for the English language. A new enriched semantic annotation of errors for this data set and new diagnostic data sets are attached in the supplementary materials.