Structured Multi-Hashing for Model Compression

TL;DR

This paper introduces a structured multi-hashing method for deep neural network compression that significantly reduces model size while maintaining or improving accuracy, enabling deployment on resource-constrained devices.

Contribution

The authors propose a novel end-to-end trainable multi-hashing technique based on matrix products, allowing direct control of model size across various architectures.

Findings

Reduced EfficientNet-B4 to B0 size with 3% accuracy gain

Compressed ResNet32 by 75% with no accuracy loss

Achieved 10x compression on CIFAR10 with over 90% accuracy

Abstract

Despite the success of deep neural networks (DNNs), state-of-the-art models are too large to deploy on low-resource devices or common server configurations in which multiple models are held in memory. Model compression methods address this limitation by reducing the memory footprint, latency, or energy consumption of a model with minimal impact on accuracy. We focus on the task of reducing the number of learnable variables in the model. In this work we combine ideas from weight hashing and dimensionality reductions resulting in a simple and powerful structured multi-hashing method based on matrix products that allows direct control of model size of any deep network and is trained end-to-end. We demonstrate the strength of our approach by compressing models from the ResNet, EfficientNet, and MobileNet architecture families. Our method allows us to drastically decrease the number of variables while maintaining high accuracy. For instance, by applying our approach to EfficentNet-B4 (16M parameters) we reduce it to to the size of B0 (5M parameters), while gaining over 3% in accuracy over B0 baseline. On the commonly used benchmark CIFAR10 we reduce the ResNet32 model by 75% with no loss in quality, and are able to do a 10x compression while still achieving above 90% accuracy.