Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognition

TL;DR

This paper introduces a Stability-Plasticity Trade-off (SPT) method for Continual Named Entity Recognition that balances knowledge retention and acquisition by combining representation pooling, weight merging, and confidence-based pseudo-labeling.

Contribution

The paper proposes a novel SPT approach that improves CNER by balancing stability and plasticity through representation and weight strategies, addressing semantic shift challenges.

Findings

SPT outperforms previous CNER methods on multiple benchmarks.

The method effectively balances knowledge retention and new learning.

Experimental results validate the approach's superiority across diverse settings.

Abstract

Continual Named Entity Recognition (CNER) is an evolving field that focuses on sequentially updating an existing model to incorporate new entity types. Previous CNER methods primarily utilize Knowledge Distillation (KD) to preserve prior knowledge and overcome catastrophic forgetting, strictly ensuring that the representations of old and new models remain consistent. Consequently, they often impart the model with excessive stability (i.e., retention of old knowledge) but limited plasticity (i.e., acquisition of new knowledge). To address this issue, we propose a Stability-Plasticity Trade-off (SPT) method for CNER that balances these aspects from both representation and weight perspectives. From the representation perspective, we introduce a pooling operation into the original KD, permitting a level of plasticity by consolidating representation dimensions. From the weight perspective, we dynamically merge the weights of old and new models, strengthening old knowledge while maintaining new knowledge. During this fusion, we implement a weight-guided selective mechanism to prioritize significant weights. Moreover, we develop a confidence-based pseudo-labeling approach for the current non-entity type, which predicts entity types using the old model to handle the semantic shift of the non-entity type, a challenge specific to CNER that has largely been ignored by previous methods. Extensive experiments across ten CNER settings on three benchmark datasets demonstrate that our SPT method surpasses previous CNER approaches, highlighting its effectiveness in achieving a suitable stability-plasticity trade-off.