BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision

TL;DR

This paper introduces BOND, a framework that enhances open-domain NER under distant supervision by leveraging pre-trained language models and a two-stage training process, significantly improving performance on benchmark datasets.

Contribution

BOND is a novel two-stage training framework that combines pre-trained language models with self-training to address noise and incompleteness in distant supervision for NER.

Findings

BOND outperforms existing methods on 5 benchmark datasets.

The two-stage training improves recall and precision.

Self-training further enhances model performance.

Abstract

We study the open-domain named entity recognition (NER) problem under distant supervision. The distant supervision, though does not require large amounts of manual annotations, yields highly incomplete and noisy distant labels via external knowledge bases. To address this challenge, we propose a new computational framework -- BOND, which leverages the power of pre-trained language models (e.g., BERT and RoBERTa) to improve the prediction performance of NER models. Specifically, we propose a two-stage training algorithm: In the first stage, we adapt the pre-trained language model to the NER tasks using the distant labels, which can significantly improve the recall and precision; In the second stage, we drop the distant labels, and propose a self-training approach to further improve the model performance. Thorough experiments on 5 benchmark datasets demonstrate the superiority of BOND over existing distantly supervised NER methods. The code and distantly labeled data have been released in https://github.com/cliang1453/BOND.