NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging

TL;DR

This paper introduces NER-BERT, a pre-trained language model specifically designed for low-resource named entity recognition, demonstrating significant improvements over existing models across multiple domains.

Contribution

The paper constructs a high-quality, large-scale NER dataset and pre-trains NER-BERT, addressing the gap in low-resource NER performance and task-specific pre-training.

Findings

NER-BERT outperforms BERT and other baselines in low-resource settings

The model shows strong performance across nine diverse domains

Entity representations visualization confirms effectiveness

Abstract

Named entity recognition (NER) models generally perform poorly when large training datasets are unavailable for low-resource domains. Recently, pre-training a large-scale language model has become a promising direction for coping with the data scarcity issue. However, the underlying discrepancies between the language modeling and NER task could limit the models' performance, and pre-training for the NER task has rarely been studied since the collected NER datasets are generally small or large but with low quality. In this paper, we construct a massive NER corpus with a relatively high quality, and we pre-train a NER-BERT model based on the created dataset. Experimental results show that our pre-trained model can significantly outperform BERT as well as other strong baselines in low-resource scenarios across nine diverse domains. Moreover, a visualization of entity representations further indicates the effectiveness of NER-BERT for categorizing a variety of entities.