Finetuning BERT on Partially Annotated NER Corpora

TL;DR

This paper proposes a method for finetuning BERT on partially annotated NER datasets using self-supervision and label preprocessing, significantly improving performance when labels are sparse.

Contribution

It introduces a novel approach that enables effective finetuning of BERT on partially labelled datasets, outperforming previous LSTM-based methods.

Findings

Outperforms previous LSTM-based label preprocessing baseline.

Achieves baseline performance with only 10% of entities labelled.

Demonstrates effective finetuning of RoBERTa on sparse annotations.

Abstract

Most Named Entity Recognition (NER) models operate under the assumption that training datasets are fully labelled. While it is valid for established datasets like CoNLL 2003 and OntoNotes, sometimes it is not feasible to obtain the complete dataset annotation. These situations may occur, for instance, after selective annotation of entities for cost reduction. This work presents an approach to finetuning BERT on such partially labelled datasets using self-supervision and label preprocessing. Our approach outperforms the previous LSTM-based label preprocessing baseline, significantly improving the performance on poorly labelled datasets. We demonstrate that following our approach while finetuning RoBERTa on CoNLL 2003 dataset with only 10% of total entities labelled is enough to reach the performance of the baseline trained on the same dataset with 50% of the entities labelled.