NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation

TL;DR

NAYER introduces a novel data-free knowledge distillation method that uses a noisy layer and label-text embeddings to generate high-quality, diverse samples efficiently, significantly outperforming existing approaches in speed and quality.

Contribution

The paper proposes NAYER, a new approach that relocates noise to a layer and uses stored label embeddings to improve sample quality and diversity in data-free knowledge distillation.

Findings

Outperforms state-of-the-art methods in sample quality.

Achieves 5 to 15 times faster training speeds.

Generates diverse samples with fewer training steps.

Abstract

Data-Free Knowledge Distillation (DFKD) has made significant recent strides by transferring knowledge from a teacher neural network to a student neural network without accessing the original data. Nonetheless, existing approaches encounter a significant challenge when attempting to generate samples from random noise inputs, which inherently lack meaningful information. Consequently, these models struggle to effectively map this noise to the ground-truth sample distribution, resulting in prolonging training times and low-quality outputs. In this paper, we propose a novel Noisy Layer Generation method (NAYER) which relocates the random source from the input to a noisy layer and utilizes the meaningful constant label-text embedding (LTE) as the input. LTE is generated by using the language model once, and then it is stored in memory for all subsequent training processes. The significance of LTE lies in its ability to contain substantial meaningful inter-class information, enabling the generation of high-quality samples with only a few training steps. Simultaneously, the noisy layer plays a key role in addressing the issue of diversity in sample generation by preventing the model from overemphasizing the constrained label information. By reinitializing the noisy layer in each iteration, we aim to facilitate the generation of diverse samples while still retaining the method's efficiency, thanks to the ease of learning provided by LTE. Experiments carried out on multiple datasets demonstrate that our NAYER not only outperforms the state-of-the-art methods but also achieves speeds 5 to 15 times faster than previous approaches. The code is available at https://github.com/tmtuan1307/nayer.