Compression strategies and space-conscious representations for deep neural networks

TL;DR

This paper explores various compression techniques for deep neural networks, including pruning, quantization, and source coding, achieving significant size reduction while maintaining or enhancing performance.

Contribution

It systematically evaluates combined compression strategies on multiple datasets, demonstrating up to 165-fold size reduction with preserved or improved accuracy.

Findings

Achieved up to 165x compression rates.

Maintained or improved model performance after compression.

Validated techniques across diverse datasets.

Abstract

Recent advances in deep learning have made available large, powerful convolutional neural networks (CNN) with state-of-the-art performance in several real-world applications. Unfortunately, these large-sized models have millions of parameters, thus they are not deployable on resource-limited platforms (e.g. where RAM is limited). Compression of CNNs thereby becomes a critical problem to achieve memory-efficient and possibly computationally faster model representations. In this paper, we investigate the impact of lossy compression of CNNs by weight pruning and quantization, and lossless weight matrix representations based on source coding. We tested several combinations of these techniques on four benchmark datasets for classification and regression problems, achieving compression rates up to $165$ times, while preserving or improving the model performance.