eTag: Class-Incremental Learning with Embedding Distillation and Task-Oriented Generation

TL;DR

The paper introduces eTag, a novel data-free class-incremental learning method that uses embedding distillation and task-oriented feature generation to prevent forgetting without storing exemplars or prototypes.

Contribution

eTag is the first to combine embedding distillation with task-oriented feature generation for data-free incremental learning.

Findings

eTag outperforms state-of-the-art methods on CIFAR-100.

eTag effectively prevents catastrophic forgetting.

eTag operates without storing exemplars or prototypes.

Abstract

Class-Incremental Learning (CIL) aims to solve the neural networks' catastrophic forgetting problem, which refers to the fact that once the network updates on a new task, its performance on previously-learned tasks drops dramatically. Most successful CIL methods incrementally train a feature extractor with the aid of stored exemplars, or estimate the feature distribution with the stored prototypes. However, the stored exemplars would violate the data privacy concerns, while the stored prototypes might not reasonably be consistent with a proper feature distribution, hindering the exploration of real-world CIL applications. In this paper, we propose a method of \textit{e}mbedding distillation and \textit{Ta}sk-oriented \textit{g}eneration (\textit{eTag}) for CIL, which requires neither the exemplar nor the prototype. Instead, eTag achieves a data-free manner to train the neural networks incrementally. To prevent the feature extractor from forgetting, eTag distills the embeddings of the network's intermediate blocks. Additionally, eTag enables a generative network to produce suitable features, fitting the needs of the top incremental classifier. Experimental results confirmed that our proposed eTag considerably outperforms the state-of-the-art methods on CIFAR-100 and ImageNet-sub\footnote{Our code is available in the Supplementary Materials.