Condensed Data Expansion Using Model Inversion for Knowledge Distillation

TL;DR

This paper introduces a novel data expansion technique using model inversion to generate synthetic data, significantly improving knowledge distillation performance with condensed datasets, especially in low-data scenarios.

Contribution

The proposed method enhances condensed datasets with synthetic data generated via model inversion, leading to improved KD accuracy and robustness in limited data settings.

Findings

Achieves up to 11.4% accuracy improvement over standard methods.

Effective even with as few as one sample per class.

Outperforms existing model inversion-based KD approaches.

Abstract

Condensed datasets offer a compact representation of larger datasets, but training models directly on them or using them to enhance model performance through knowledge distillation (KD) can result in suboptimal outcomes due to limited information. To address this, we propose a method that expands condensed datasets using model inversion, a technique for generating synthetic data based on the impressions of a pre-trained model on its training data. This approach is particularly well-suited for KD scenarios, as the teacher model is already pre-trained and retains knowledge of the original training data. By creating synthetic data that complements the condensed samples, we enrich the training set and better approximate the underlying data distribution, leading to improvements in student model accuracy during knowledge distillation. Our method demonstrates significant gains in KD accuracy compared to using condensed datasets alone and outperforms standard model inversion-based KD methods by up to 11.4% across various datasets and model architectures. Importantly, it remains effective even when using as few as one condensed sample per class, and can also enhance performance in few-shot scenarios where only limited real data samples are available.