Decomposable-Net: Scalable Low-Rank Compression for Neural Networks

TL;DR

Decomposable-Net enables flexible neural network size adjustment via low-rank decomposition without retraining, maintaining or improving performance across sizes, demonstrated on ImageNet with superior accuracy and efficiency.

Contribution

We introduce a novel backpropagation method for low-rank decomposition that allows neural networks to be scaled to any size without retraining, unlike previous fixed-size methods.

Findings

Achieves 73.2% top-1 accuracy with 0.27x MACs on ResNet-50.

Outperforms Tucker decomposition, Trained Rank Pruning, and universally slimmable networks.

Effectively suppresses approximation error with a new rank selection criterion.

Abstract

Compressing DNNs is important for the real-world applications operating on resource-constrained devices. However, we typically observe drastic performance deterioration when changing model size after training is completed. Therefore, retraining is required to resume the performance of the compressed models suitable for different devices. In this paper, we propose Decomposable-Net (the network decomposable in any size), which allows flexible changes to model size without retraining. We decompose weight matrices in the DNNs via singular value decomposition and adjust ranks according to the target model size. Unlike the existing low-rank compression methods that specialize the model to a fixed size, we propose a novel backpropagation scheme that jointly minimizes losses for both of full- and low-rank networks. This enables not only to maintain the performance of a full-rank network {\it without retraining} but also to improve low-rank networks in multiple sizes. Additionally, we introduce a simple criterion for rank selection that effectively suppresses approximation error. In experiments on the ImageNet classification task, Decomposable-Net yields superior accuracy in a wide range of model sizes. In particular, Decomposable-Net achieves the top-1 accuracy of $73.2\%$ with $0.27\times$MACs with ResNet-50, compared to Tucker decomposition ($67.4\% / 0.30\times$), Trained Rank Pruning ($70.6\% / 0.28\times$), and universally slimmable networks ($71.4\% / 0.26\times$).