Unified Data-Free Compression: Pruning and Quantization without Fine-Tuning

TL;DR

This paper introduces a unified data-free framework for neural network pruning and quantization that performs both simultaneously without data or fine-tuning, restoring information loss through a theoretically derived reconstruction method.

Contribution

The novel UDFC framework enables combined pruning and quantization without data or fine-tuning, leveraging a linear combination assumption for channel information preservation.

Findings

Achieves 20.54% accuracy improvement on ImageNet over SOTA.

Supports various architectures and compression methods.

Effective with 30% pruning and 6-bit quantization.

Abstract

Structured pruning and quantization are promising approaches for reducing the inference time and memory footprint of neural networks. However, most existing methods require the original training dataset to fine-tune the model. This not only brings heavy resource consumption but also is not possible for applications with sensitive or proprietary data due to privacy and security concerns. Therefore, a few data-free methods are proposed to address this problem, but they perform data-free pruning and quantization separately, which does not explore the complementarity of pruning and quantization. In this paper, we propose a novel framework named Unified Data-Free Compression(UDFC), which performs pruning and quantization simultaneously without any data and fine-tuning process. Specifically, UDFC starts with the assumption that the partial information of a damaged(e.g., pruned or quantized) channel can be preserved by a linear combination of other channels, and then derives the reconstruction form from the assumption to restore the information loss due to compression. Finally, we formulate the reconstruction error between the original network and its compressed network, and theoretically deduce the closed-form solution. We evaluate the UDFC on the large-scale image classification task and obtain significant improvements over various network architectures and compression methods. For example, we achieve a 20.54% accuracy improvement on ImageNet dataset compared to SOTA method with 30% pruning ratio and 6-bit quantization on ResNet-34.