Towards Lingua Franca Named Entity Recognition with BERT

TL;DR

This paper presents a multilingual BERT-based Named Entity Recognition model trained on multiple languages simultaneously, achieving state-of-the-art results and enabling zero-shot predictions on unseen languages.

Contribution

It introduces a single multilingual NER model trained jointly on many languages, improving accuracy and zero-shot capabilities compared to language-specific models.

Findings

Achieves state-of-the-art results on Dutch, Spanish, Arabic, and Chinese datasets.

Performs competitively with monolingual models across multiple languages.

Demonstrates effective zero-shot NER on unseen languages.

Abstract

Information extraction is an important task in NLP, enabling the automatic extraction of data for relational database filling. Historically, research and data was produced for English text, followed in subsequent years by datasets in Arabic, Chinese (ACE/OntoNotes), Dutch, Spanish, German (CoNLL evaluations), and many others. The natural tendency has been to treat each language as a different dataset and build optimized models for each. In this paper we investigate a single Named Entity Recognition model, based on a multilingual BERT, that is trained jointly on many languages simultaneously, and is able to decode these languages with better accuracy than models trained only on one language. To improve the initial model, we study the use of regularization strategies such as multitask learning and partial gradient updates. In addition to being a single model that can tackle multiple languages (including code switch), the model could be used to make zero-shot predictions on a new language, even ones for which training data is not available, out of the box. The results show that this model not only performs competitively with monolingual models, but it also achieves state-of-the-art results on the CoNLL02 Dutch and Spanish datasets, OntoNotes Arabic and Chinese datasets. Moreover, it performs reasonably well on unseen languages, achieving state-of-the-art for zero-shot on three CoNLL languages.