Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay

TL;DR

This paper introduces data-free generative replay and dual-teacher knowledge distillation techniques to improve class-incremental learning, reducing reliance on pre-trained generative models and enhancing knowledge transfer, demonstrated on CIFAR-100 and ImageNet.

Contribution

It presents novel methods for data-free generative replay and dual-teacher distillation, advancing class-incremental learning without pre-trained generative models.

Findings

Improved accuracy on CIFAR-100 and ImageNet datasets.

Reduced memory and training costs compared to traditional methods.

Enhanced knowledge transfer in incremental learning scenarios.

Abstract

This paper proposes two novel knowledge transfer techniques for class-incremental learning (CIL). First, we propose data-free generative replay (DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic samples from a generative model. In the conventional generative replay, the generative model is pre-trained for old data and shared in extra memory for later incremental learning. In our proposed DF-GR, we train a generative model from scratch without using any training data, based on the pre-trained classification model from the past, so we curtail the cost of sharing pre-trained generative models. Second, we introduce dual-teacher information distillation (DT-ID) for knowledge distillation from two teachers to one student. In CIL, we use DT-ID to learn new classes incrementally based on the pre-trained model for old classes and another model (pre-)trained on the new data for new classes. We implemented the proposed schemes on top of one of the state-of-the-art CIL methods and showed the performance improvement on CIFAR-100 and ImageNet datasets.