MultiCoNER: A Large-scale Multilingual dataset for Complex Named Entity Recognition

TL;DR

MultiCoNER introduces a comprehensive multilingual dataset for Named Entity Recognition, covering diverse languages, domains, and challenging scenarios, to facilitate the development of more robust NER systems.

Contribution

It provides a large-scale, multilingual NER dataset with challenging scenarios, and evaluates models including a gazetteer-enhanced approach, highlighting current limitations.

Findings

Baseline XLM-RoBERTa achieves 54% macro-F1 on the dataset.

GEMNET with gazetteers improves performance by 30% macro-F1.

The dataset exposes challenges for large pre-trained models in NER.

Abstract

We present MultiCoNER, a large multilingual dataset for Named Entity Recognition that covers 3 domains (Wiki sentences, questions, and search queries) across 11 languages, as well as multilingual and code-mixing subsets. This dataset is designed to represent contemporary challenges in NER, including low-context scenarios (short and uncased text), syntactically complex entities like movie titles, and long-tail entity distributions. The 26M token dataset is compiled from public resources using techniques such as heuristic-based sentence sampling, template extraction and slotting, and machine translation. We applied two NER models on our dataset: a baseline XLM-RoBERTa model, and a state-of-the-art GEMNET model that leverages gazetteers. The baseline achieves moderate performance (macro-F1=54%), highlighting the difficulty of our data. GEMNET, which uses gazetteers, improvement significantly (average improvement of macro-F1=+30%). MultiCoNER poses challenges even for large pre-trained language models, and we believe that it can help further research in building robust NER systems. MultiCoNER is publicly available at https://registry.opendata.aws/multiconer/ and we hope that this resource will help advance research in various aspects of NER.