Improving NER's Performance with Massive financial corpus

TL;DR

This paper presents a method to improve Named Entity Recognition (NER) performance in financial texts by pre-training lightweight models with knowledge distillation and multi-stage learning, achieving higher recall and faster training.

Contribution

It introduces a novel approach combining pre-training, knowledge distillation, and multi-stage learning to enhance NER in financial domains with limited data.

Findings

Recall increased by nearly 20 points

Training speed was four times faster than BERT-CRF

Effective in low-quality, small-scale datasets

Abstract

Training large deep neural networks needs massive high quality annotation data, but the time and labor costs are too expensive for small business. We start a company-name recognition task with a small scale and low quality training data, then using skills to enhanced model training speed and predicting performance with minimum labor cost. The methods we use involve pre-training a lite language model such as Albert-small or Electra-small in financial corpus, knowledge of distillation and multi-stage learning. The result is that we raised the recall rate by nearly 20 points and get 4 times as fast as BERT-CRF model.