Class-Incremental Learning based on Label Generation

TL;DR

This paper introduces VAG, a class-incremental learning method that reformulates the task as label generation, reducing catastrophic forgetting and improving retention of pre-trained model representations.

Contribution

It proposes a novel CIL approach leveraging label generation and vocabulary sparsity, significantly outperforming existing baselines.

Findings

VAG reduces catastrophic forgetting in CIL.

VAG outperforms baseline methods by a large margin.

Reformulating CIL as label generation enhances model retention.

Abstract

Despite the great success of pre-trained language models, it is still a challenge to use these models for continual learning, especially for the class-incremental learning (CIL) setting due to catastrophic forgetting (CF). This paper reports our finding that if we formulate CIL as a continual label generation problem, CF is drastically reduced and the generalizable representations of pre-trained models can be better retained. We thus propose a new CIL method (VAG) that also leverages the sparsity of vocabulary to focus the generation and creates pseudo-replay samples by using label semantics. Experimental results show that VAG outperforms baselines by a large margin.