Prompt Tuning for Few-Shot Continual Learning Named Entity Recognition

TL;DR

This paper introduces a prompt tuning approach with memory demonstration templates to improve few-shot continual learning for named entity recognition, addressing knowledge distillation challenges and enhancing performance.

Contribution

The paper proposes an expandable Anchor words-oriented Prompt Tuning paradigm and Memory Demonstration Templates to tackle few-shot CLNER challenges and improve model generalization.

Findings

Achieves competitive performance on FS-CLNER tasks.

Effectively mitigates the Few-Shot Distillation Dilemma.

Enhances in-context learning with memory templates.

Abstract

Knowledge distillation has been successfully applied to Continual Learning Named Entity Recognition (CLNER) tasks, by using a teacher model trained on old-class data to distill old-class entities present in new-class data as a form of regularization, thereby avoiding catastrophic forgetting. However, in Few-Shot CLNER (FS-CLNER) tasks, the scarcity of new-class entities makes it difficult for the trained model to generalize during inference. More critically, the lack of old-class entity information hinders the distillation of old knowledge, causing the model to fall into what we refer to as the Few-Shot Distillation Dilemma. In this work, we address the above challenges through a prompt tuning paradigm and memory demonstration template strategy. Specifically, we designed an expandable Anchor words-oriented Prompt Tuning (APT) paradigm to bridge the gap between pre-training and fine-tuning, thereby enhancing performance in few-shot scenarios. Additionally, we incorporated Memory Demonstration Templates (MDT) into each training instance to provide replay samples from previous tasks, which not only avoids the Few-Shot Distillation Dilemma but also promotes in-context learning. Experiments show that our approach achieves competitive performances on FS-CLNER.