Task Relation Distillation and Prototypical Pseudo Label for Incremental Named Entity Recognition

TL;DR

This paper introduces RDP, a novel method for incremental NER that reduces catastrophic forgetting and background shift by using task relation distillation and prototypical pseudo labels, significantly improving performance.

Contribution

The paper proposes a new approach combining task relation distillation and prototypical pseudo labeling to enhance incremental NER, addressing key challenges of forgetting and background shift.

Findings

Achieves 6.08% higher Micro F1 score on average

Achieves 7.71% higher Macro F1 score on average

Outperforms previous state-of-the-art methods on multiple datasets

Abstract

Incremental Named Entity Recognition (INER) involves the sequential learning of new entity types without accessing the training data of previously learned types. However, INER faces the challenge of catastrophic forgetting specific for incremental learning, further aggravated by background shift (i.e., old and future entity types are labeled as the non-entity type in the current task). To address these challenges, we propose a method called task Relation Distillation and Prototypical pseudo label (RDP) for INER. Specifically, to tackle catastrophic forgetting, we introduce a task relation distillation scheme that serves two purposes: 1) ensuring inter-task semantic consistency across different incremental learning tasks by minimizing inter-task relation distillation loss, and 2) enhancing the model's prediction confidence by minimizing intra-task self-entropy loss. Simultaneously, to mitigate background shift, we develop a prototypical pseudo label strategy that distinguishes old entity types from the current non-entity type using the old model. This strategy generates high-quality pseudo labels by measuring the distances between token embeddings and type-wise prototypes. We conducted extensive experiments on ten INER settings of three benchmark datasets (i.e., CoNLL2003, I2B2, and OntoNotes5). The results demonstrate that our method achieves significant improvements over the previous state-of-the-art methods, with an average increase of 6.08% in Micro F1 score and 7.71% in Macro F1 score.