Enhancing Continual Relation Extraction via Classifier Decomposition

TL;DR

This paper introduces a classifier decomposition method to improve continual relation extraction by reducing biases, leading to better retention of old knowledge and more robust learning of new relations.

Contribution

It proposes a novel classifier decomposition framework that splits the last FFN layer to address biases in CRE models, outperforming previous state-of-the-art methods.

Findings

Consistently outperforms existing CRE models on benchmarks.

Effectively reduces classifier and representation biases.

Highlights the importance of initial training stages in CRE.

Abstract

Continual relation extraction (CRE) models aim at handling emerging new relations while avoiding catastrophically forgetting old ones in the streaming data. Though improvements have been shown by previous CRE studies, most of them only adopt a vanilla strategy when models first learn representations of new relations. In this work, we point out that there exist two typical biases after training of this vanilla strategy: classifier bias and representation bias, which causes the previous knowledge that the model learned to be shaded. To alleviate those biases, we propose a simple yet effective classifier decomposition framework that splits the last FFN layer into separated previous and current classifiers, so as to maintain previous knowledge and encourage the model to learn more robust representations at this training stage. Experimental results on two standard benchmarks show that our proposed framework consistently outperforms the state-of-the-art CRE models, which indicates that the importance of the first training stage to CRE models may be underestimated. Our code is available at https://github.com/hemingkx/CDec.