Continual Named Entity Recognition without Catastrophic Forgetting

TL;DR

This paper presents a novel continual learning method for Named Entity Recognition that effectively mitigates catastrophic forgetting and semantic shift issues through pooled feature distillation, confidence-based pseudo-labeling, and adaptive re-weighting, outperforming previous approaches.

Contribution

The paper introduces a pooled feature distillation loss, confidence-based pseudo-labeling, and an adaptive re-weighting strategy to improve continual NER performance and address semantic shift and class imbalance.

Findings

Significantly outperforms prior state-of-the-art methods.

Achieves an average of 6.3% and 8.0% improvements in Micro and Macro F1 scores.

Validated on ten CNER settings across three datasets.

Abstract

Continual Named Entity Recognition (CNER) is a burgeoning area, which involves updating an existing model by incorporating new entity types sequentially. Nevertheless, continual learning approaches are often severely afflicted by catastrophic forgetting. This issue is intensified in CNER due to the consolidation of old entity types from previous steps into the non-entity type at each step, leading to what is known as the semantic shift problem of the non-entity type. In this paper, we introduce a pooled feature distillation loss that skillfully navigates the trade-off between retaining knowledge of old entity types and acquiring new ones, thereby more effectively mitigating the problem of catastrophic forgetting. Additionally, we develop a confidence-based pseudo-labeling for the non-entity type, \emph{i.e.,} predicting entity types using the old model to handle the semantic shift of the non-entity type. Following the pseudo-labeling process, we suggest an adaptive re-weighting type-balanced learning strategy to handle the issue of biased type distribution. We carried out comprehensive experiments on ten CNER settings using three different datasets. The results illustrate that our method significantly outperforms prior state-of-the-art approaches, registering an average improvement of $6.3$\% and $8.0$\% in Micro and Macro F1 scores, respectively.