Data-Free Adversarial Distillation

TL;DR

This paper introduces a novel data-free adversarial distillation method that enables model compression without relying on original training data, achieving competitive results and extending to complex tasks like semantic segmentation.

Contribution

The paper proposes a new data-free adversarial distillation technique using a model discrepancy measure and a generator-discriminator setup, enabling effective knowledge transfer without real data.

Findings

Achieves comparable performance to data-driven methods.

Extends to semantic segmentation with state-of-the-art results.

Demonstrates the effectiveness of adversarial sample generation for model distillation.

Abstract

Knowledge Distillation (KD) has made remarkable progress in the last few years and become a popular paradigm for model compression and knowledge transfer. However, almost all existing KD algorithms are data-driven, i.e., relying on a large amount of original training data or alternative data, which is usually unavailable in real-world scenarios. In this paper, we devote ourselves to this challenging problem and propose a novel adversarial distillation mechanism to craft a compact student model without any real-world data. We introduce a model discrepancy to quantificationally measure the difference between student and teacher models and construct an optimizable upper bound. In our work, the student and the teacher jointly act the role of the discriminator to reduce this discrepancy, when a generator adversarially produces some "hard samples" to enlarge it. Extensive experiments demonstrate that the proposed data-free method yields comparable performance to existing data-driven methods. More strikingly, our approach can be directly extended to semantic segmentation, which is more complicated than classification, and our approach achieves state-of-the-art results. Code and pretrained models are available at https://github.com/VainF/Data-Free-Adversarial-Distillation.