Conditional Generative Data-free Knowledge Distillation

TL;DR

This paper introduces a data-free knowledge distillation method using conditional image generation, enabling training lightweight models without access to original data, and achieves state-of-the-art results on multiple datasets.

Contribution

It proposes a novel conditional generative framework for data-free knowledge distillation, combining semi-supervised training and adversarial loss functions.

Findings

Achieves state-of-the-art results on CIFAR100, Caltech101, and ImageNet datasets.

Effectively generates class-specific training images without original data.

Enhances student model performance through a novel adversarial training framework.

Abstract

Knowledge distillation has made remarkable achievements in model compression. However, most existing methods require the original training data, which is usually unavailable due to privacy and security issues. In this paper, we propose a conditional generative data-free knowledge distillation (CGDD) framework for training lightweight networks without any training data. This method realizes efficient knowledge distillation based on conditional image generation. Specifically, we treat the preset labels as ground truth to train a conditional generator in a semi-supervised manner. The trained generator can produce specified classes of training images. For training the student network, we force it to extract the knowledge hidden in teacher feature maps, which provide crucial cues for the learning process. Moreover, an adversarial training framework for promoting distillation performance is constructed by designing several loss functions. This framework helps the student model to explore larger data space. To demonstrate the effectiveness of the proposed method, we conduct extensive experiments on different datasets. Compared with other data-free works, our work obtains state-of-the-art results on CIFAR100, Caltech101, and different versions of ImageNet datasets. The codes will be released.