Joint Matrix Decomposition for Deep Convolutional Neural Networks Compression

TL;DR

This paper introduces a joint matrix decomposition approach to compress deep CNNs, significantly reducing model size while maintaining accuracy better than existing methods.

Contribution

It proposes a novel joint decomposition technique that compresses CNN layers collectively, improving compression ratios and reducing performance loss compared to prior layer-wise methods.

Findings

ResNet-34 compressed by 22X with minimal accuracy loss

Outperforms state-of-the-art compression methods

Effective across multiple CNN architectures and datasets

Abstract

Deep convolutional neural networks (CNNs) with a large number of parameters require intensive computational resources, and thus are hard to be deployed in resource-constrained platforms. Decomposition-based methods, therefore, have been utilized to compress CNNs in recent years. However, since the compression factor and performance are negatively correlated, the state-of-the-art works either suffer from severe performance degradation or have relatively low compression factors. To overcome this problem, we propose to compress CNNs and alleviate performance degradation via joint matrix decomposition, which is different from existing works that compressed layers separately. The idea is inspired by the fact that there are lots of repeated modules in CNNs. By projecting weights with the same structures into the same subspace, networks can be jointly compressed with larger ranks. In particular, three joint matrix decomposition schemes are developed, and the corresponding optimization approaches based on Singular Value Decomposition are proposed. Extensive experiments are conducted across three challenging compact CNNs for different benchmark data sets to demonstrate the superior performance of our proposed algorithms. As a result, our methods can compress the size of ResNet-34 by 22X with slighter accuracy degradation compared with several state-of-the-art methods.