Coarse-to-Fine Pre-training for Named Entity Recognition

TL;DR

This paper introduces a NER-specific pre-training framework that progressively injects coarse-to-fine entity knowledge into models, significantly improving performance on multiple benchmarks without relying on labeled data.

Contribution

It proposes a novel coarse-to-fine pre-training approach for NER that leverages automatically mined entity knowledge at different granularities, enhancing existing models.

Findings

Achieves state-of-the-art results on three NER benchmarks.

Improves performance in low-resource and label-few scenarios.

Demonstrates effectiveness without human-labeled data.

Abstract

More recently, Named Entity Recognition hasachieved great advances aided by pre-trainingapproaches such as BERT. However, currentpre-training techniques focus on building lan-guage modeling objectives to learn a gen-eral representation, ignoring the named entity-related knowledge. To this end, we proposea NER-specific pre-training framework to in-ject coarse-to-fine automatically mined entityknowledge into pre-trained models. Specifi-cally, we first warm-up the model via an en-tity span identification task by training it withWikipedia anchors, which can be deemed asgeneral-typed entities. Then we leverage thegazetteer-based distant supervision strategy totrain the model extract coarse-grained typedentities. Finally, we devise a self-supervisedauxiliary task to mine the fine-grained namedentity knowledge via clustering.Empiricalstudies on three public NER datasets demon-strate that our framework achieves significantimprovements against several pre-trained base-lines, establishing the new state-of-the-art per-formance on three benchmarks. Besides, weshow that our framework gains promising re-sults without using human-labeled trainingdata, demonstrating its effectiveness in label-few and low-resource scenarios