C2G-KD: PCA-Constrained Generator for Data-Free Knowledge Distillation

TL;DR

C2G-KD is a data-free knowledge distillation method that uses a PCA-constrained generator to produce synthetic data guided by a teacher model, enabling effective training without real data.

Contribution

It introduces a PCA-based geometric constraint for synthetic data generation in data-free knowledge distillation, improving diversity and topological consistency.

Findings

Effective on MNIST with minimal class data

Preserves topological structure of classes

Generates useful synthetic samples for distillation

Abstract

We introduce C2G-KD, a data-free knowledge distillation framework where a class-conditional generator is trained to produce synthetic samples guided by a frozen teacher model and geometric constraints derived from PCA. The generator never observes real training data but instead learns to activate the teacher's output through a combination of semantic and structural losses. By constraining generated samples to lie within class-specific PCA subspaces estimated from as few as two real examples per class, we preserve topological consistency and diversity. Experiments on MNIST show that even minimal class structure is sufficient to bootstrap useful synthetic training pipelines.