Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data

TL;DR

This paper introduces a data-free knowledge distillation framework that dynamically maintains synthetic samples and matches real data distribution to prevent catastrophic forgetting and distribution mismatch, improving student model accuracy.

Contribution

It proposes a novel data-free KD method with dynamic sample collection and distribution matching, addressing catastrophic forgetting and data mismatch issues in model compression.

Findings

Improved student model accuracy over state-of-the-art methods

Effective prevention of catastrophic forgetting in data-free KD

Enhanced synthetic data generation matching real data distribution

Abstract

With the increasing popularity of deep learning on edge devices, compressing large neural networks to meet the hardware requirements of resource-constrained devices became a significant research direction. Numerous compression methodologies are currently being used to reduce the memory sizes and energy consumption of neural networks. Knowledge distillation (KD) is among such methodologies and it functions by using data samples to transfer the knowledge captured by a large model (teacher) to a smaller one(student). However, due to various reasons, the original training data might not be accessible at the compression stage. Therefore, data-free model compression is an ongoing research problem that has been addressed by various works. In this paper, we point out that catastrophic forgetting is a problem that can potentially be observed in existing data-free distillation methods. Moreover, the sample generation strategies in some of these methods could result in a mismatch between the synthetic and real data distributions. To prevent such problems, we propose a data-free KD framework that maintains a dynamic collection of generated samples over time. Additionally, we add the constraint of matching the real data distribution in sample generation strategies that target maximum information gain. Our experiments demonstrate that we can improve the accuracy of the student models obtained via KD when compared with state-of-the-art approaches on the SVHN, Fashion MNIST and CIFAR100 datasets.