InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective

TL;DR

This paper introduces InfoCL, a novel replay-based continual text classification method that uses contrastive learning and adversarial memory augmentation to reduce catastrophic forgetting and improve performance on class-incremental tasks.

Contribution

The paper proposes InfoCL, a new method combining contrastive learning and adversarial memory augmentation to better preserve representations and mitigate forgetting in continual text classification.

Findings

InfoCL outperforms existing methods on three text classification tasks.

Contrastive learning enhances representation retention in continual learning.

Adversarial memory augmentation reduces overfitting of replay data.

Abstract

Continual learning (CL) aims to constantly learn new knowledge over time while avoiding catastrophic forgetting on old tasks. We focus on continual text classification under the class-incremental setting. Recent CL studies have identified the severe performance decrease on analogous classes as a key factor for catastrophic forgetting. In this paper, through an in-depth exploration of the representation learning process in CL, we discover that the compression effect of the information bottleneck leads to confusion on analogous classes. To enable the model learn more sufficient representations, we propose a novel replay-based continual text classification method, InfoCL. Our approach utilizes fast-slow and current-past contrastive learning to perform mutual information maximization and better recover the previously learned representations. In addition, InfoCL incorporates an adversarial memory augmentation strategy to alleviate the overfitting problem of replay. Experimental results demonstrate that InfoCL effectively mitigates forgetting and achieves state-of-the-art performance on three text classification tasks. The code is publicly available at https://github.com/Yifan-Song793/InfoCL.