Customizing Synthetic Data for Data-Free Student Learning

TL;DR

This paper introduces a method for customizing synthetic data in data-free knowledge distillation, dynamically adjusting data difficulty based on the student's learning ability to improve training effectiveness.

Contribution

It proposes an adaptive synthetic data generation technique using self-supervised tasks to enhance data-free student learning.

Findings

Improved student model performance across datasets

Effective data synthesis tailored to student ability

Demonstrated robustness on various models

Abstract

Data-free knowledge distillation (DFKD) aims to obtain a lightweight student model without original training data. Existing works generally synthesize data from the pre-trained teacher model to replace the original training data for student learning. To more effectively train the student model, the synthetic data shall be customized to the current student learning ability. However, this is ignored in the existing DFKD methods and thus negatively affects the student training. To address this issue, we propose Customizing Synthetic Data for Data-Free Student Learning (CSD) in this paper, which achieves adaptive data synthesis using a self-supervised augmented auxiliary task to estimate the student learning ability. Specifically, data synthesis is dynamically adjusted to enlarge the cross entropy between the labels and the predictions from the self-supervised augmented task, thus generating hard samples for the student model. The experiments on various datasets and teacher-student models show the effectiveness of our proposed method. Code is available at: $\href{https://github.com/luoshiya/CSD}{https://github.com/luoshiya/CSD}$