Less is More: Rethinking State-of-the-art Continual Relation Extraction Models with a Frustratingly Easy but Effective Approach

TL;DR

This paper introduces a simple yet effective two-stage training approach for continual relation extraction that improves performance by addressing data imbalance and decision boundary skewness.

Contribution

The proposed FEA method simplifies CRE training with two stages, achieving comparable or better results than complex state-of-the-art models.

Findings

FEA outperforms existing baselines on FewRel and TACRED datasets.

Data imbalance causes skewed decision boundaries in CRE models.

Two strong CRE baselines can be unified under the FEA training pipeline.

Abstract

Continual relation extraction (CRE) requires the model to continually learn new relations from class-incremental data streams. In this paper, we propose a Frustratingly easy but Effective Approach (FEA) method with two learning stages for CRE: 1) Fast Adaption (FA) warms up the model with only new data. 2) Balanced Tuning (BT) finetunes the model on the balanced memory data. Despite its simplicity, FEA achieves comparable (on TACRED or superior (on FewRel) performance compared with the state-of-the-art baselines. With careful examinations, we find that the data imbalance between new and old relations leads to a skewed decision boundary in the head classifiers over the pretrained encoders, thus hurting the overall performance. In FEA, the FA stage unleashes the potential of memory data for the subsequent finetuning, while the BT stage helps establish a more balanced decision boundary. With a unified view, we find that two strong CRE baselines can be subsumed into the proposed training pipeline. The success of FEA also provides actionable insights and suggestions for future model designing in CRE.