The Knowledge Within: Methods for Data-Free Model Compression

TL;DR

This paper introduces three methods to generate synthetic data from trained models, enabling data-free calibration and fine-tuning of compressed neural networks, which is crucial when real data is unavailable or sensitive.

Contribution

The paper proposes novel data-free techniques for model calibration and fine-tuning that do not require access to original training data, leveraging batch normalization statistics.

Findings

Synthetic samples enable effective model calibration without real data

The best method achieves negligible accuracy loss compared to using real data

Approach facilitates data-free deployment of compressed neural networks

Abstract

Recently, an extensive amount of research has been focused on compressing and accelerating Deep Neural Networks (DNN). So far, high compression rate algorithms require part of the training dataset for a low precision calibration, or a fine-tuning process. However, this requirement is unacceptable when the data is unavailable or contains sensitive information, as in medical and biometric use-cases. We present three methods for generating synthetic samples from trained models. Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process. Our best performing method has a negligible accuracy degradation compared to the original training set. This method, which leverages intrinsic batch normalization layers' statistics of the trained model, can be used to evaluate data similarity. Our approach opens a path towards genuine data-free model compression, alleviating the need for training data during model deployment.