Single-path Bit Sharing for Automatic Loss-aware Model Compression

TL;DR

This paper introduces Single-path Bit Sharing (SBS), a unified and efficient method for joint network pruning and quantization that automatically determines optimal compression configurations, significantly reducing computational cost while maintaining accuracy.

Contribution

SBS unifies pruning and quantization into a single model with learnable binary gates, enabling automatic configuration search with reduced complexity and cost.

Findings

Achieves 22.6x BOP reduction on MobileNetV2 with 0.1% accuracy drop

Outperforms existing methods in efficiency and compression ratio

Effective on CIFAR-100 and ImageNet datasets

Abstract

Network pruning and quantization are proven to be effective ways for deep model compression. To obtain a highly compact model, most methods first perform network pruning and then conduct network quantization based on the pruned model. However, this strategy may ignore that they would affect each other and thus performing them separately may lead to sub-optimal performance. To address this, performing pruning and quantization jointly is essential. Nevertheless, how to make a trade-off between pruning and quantization is non-trivial. Moreover, existing compression methods often rely on some pre-defined compression configurations. Some attempts have been made to search for optimal configurations, which however may take unbearable optimization cost. To address the above issues, we devise a simple yet effective method named Single-path Bit Sharing (SBS). Specifically, we first consider network pruning as a special case of quantization, which provides a unified view for pruning and quantization. We then introduce a single-path model to encode all candidate compression configurations. In this way, the configuration search problem is transformed into a subset selection problem, which significantly reduces the number of parameters, computational cost and optimization difficulty. Relying on the single-path model, we further introduce learnable binary gates to encode the choice of bitwidth. By jointly training the binary gates in conjunction with network parameters, the compression configurations of each layer can be automatically determined. Extensive experiments on both CIFAR-100 and ImageNet show that SBS is able to significantly reduce computational cost while achieving promising performance. For example, our SBS compressed MobileNetV2 achieves 22.6x Bit-Operation (BOP) reduction with only 0.1% drop in the Top-1 accuracy.