What Makes a Good Dataset for Knowledge Distillation?

TL;DR

This paper investigates what characteristics make a dataset suitable for effective knowledge distillation, showing that various types of datasets, including synthetic data, can be used successfully for model compression.

Contribution

The study systematically evaluates different surrogate datasets for knowledge distillation and identifies key criteria that determine their effectiveness.

Findings

Synthetic and alternative datasets can effectively replace original data in KD.

Certain dataset properties, such as diversity and relevance, influence distillation success.

Many datasets beyond real in-domain images are viable for KD.

Abstract

Knowledge distillation (KD) has been a popular and effective method for model compression. One important assumption of KD is that the teacher's original dataset will also be available when training the student. However, in situations such as continual learning and distilling large models trained on company-withheld datasets, having access to the original data may not always be possible. This leads practitioners towards utilizing other sources of supplemental data, which could yield mixed results. One must then ask: "what makes a good dataset for transferring knowledge from teacher to student?" Many would assume that only real in-domain imagery is viable, but is that the only option? In this work, we explore multiple possible surrogate distillation datasets and demonstrate that many different datasets, even unnatural synthetic imagery, can serve as a suitable alternative in KD. From examining these alternative datasets, we identify and present various criteria describing what makes a good dataset for distillation. Source code is available at https://github.com/osu-cvl/good-kd-dataset.