Dream Distillation: A Data-Independent Model Compression Framework

TL;DR

Dream Distillation introduces a novel data-independent model compression method that effectively compresses models without access to original or alternative datasets, enabling deployment in privacy-sensitive scenarios.

Contribution

It presents the first framework for model compression that does not require any real data, addressing privacy and data access issues in deep learning deployment.

Findings

Achieves 88.5% accuracy on CIFAR-10 without original data

Demonstrates effectiveness in privacy-sensitive model deployment

Provides a new direction for data-independent model compression

Abstract

Model compression is eminently suited for deploying deep learning on IoT-devices. However, existing model compression techniques rely on access to the original or some alternate dataset. In this paper, we address the model compression problem when no real data is available, e.g., when data is private. To this end, we propose Dream Distillation, a data-independent model compression framework. Our experiments show that Dream Distillation can achieve 88.5% accuracy on the CIFAR-10 test set without actually training on the original data!