Diverse Image Priors for Black-box Data-free Knowledge Distillation

TL;DR

DIP-KD is a novel framework for black-box, data-free knowledge distillation that leverages diverse image priors, contrastive learning, and a primer student to improve model transfer without access to original data.

Contribution

It introduces a three-phase collaborative pipeline that synthesizes diverse image priors, enhances synthetic data distinction, and employs a primer student for effective knowledge transfer.

Findings

Achieves state-of-the-art results across 12 benchmarks.

Data diversity is critical for effective knowledge distillation in restricted environments.

Contrastive learning enhances the distinction of synthetic samples.

Abstract

Knowledge distillation (KD) represents a vital mechanism to transfer expertise from complex teacher networks to efficient student models. However, in decentralized or secure AI ecosystems, privacy regulations and proprietary interests often restrict access to the teacher's interface and original datasets. These constraints define a challenging black-box data-free KD scenario where only top-1 predictions and no training data are available. While recent approaches utilize synthetic data, they still face limitations in data diversity and distillation signals. We propose Diverse Image Priors Knowledge Distillation (DIP-KD), a framework that addresses these challenges through a three-phase collaborative pipeline: (1) Synthesis of image priors to capture diverse visual patterns and semantics; (2) Contrast to enhance the collective distinction between synthetic samples via contrastive learning; and (3) Distillation via a novel primer student that enables soft-probability KD. Our evaluation across 12 benchmarks shows that DIP-KD achieves state-of-the-art performance, with ablations confirming data diversity as critical for knowledge acquisition in restricted AI environments.