DeepTwist: Learning Model Compression via Occasional Weight Distortion

TL;DR

DeepTwist introduces a simple, efficient framework for model compression that distorts weights occasionally, significantly improving compression rates across various techniques with minimal additional effort.

Contribution

It proposes a novel weight distortion method that enhances compression efficiency without altering existing training algorithms.

Findings

Improves compression rates for pruning, quantization, and low-rank approximation.

Reduces need for retraining and hyper-parameter tuning.

Provides regularization benefits.

Abstract

Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model compression is, however, still challenging because the design complexity of model compression is rapidly increasing due to additional hyper-parameters and computation overhead in order to achieve a high compression ratio. In this paper, we propose a simple and efficient model compression framework called DeepTwist which distorts weights in an occasional manner without modifying the underlying training algorithms. The ideas of designing weight distortion functions are intuitive and straightforward given formats of compressed weights. We show that our proposed framework improves compression rate significantly for pruning, quantization, and low-rank approximation techniques while the efforts of additional retraining and/or hyper-parameter search are highly reduced. Regularization effects of DeepTwist are also reported.