Convolutional Neural Network Compression Based on Low-Rank Decomposition

TL;DR

This paper introduces a neural network compression technique combining Variational Bayesian Matrix Factorization with orthogonal regularization to reduce model size while maintaining accuracy, suitable for edge device deployment.

Contribution

It presents a novel integration of VBMF and orthogonal regularization for CNN compression, improving accuracy retention at various compression ratios.

Findings

Outperforms existing compression methods at high and low ratios

Maintains high accuracy with significant model size reduction

Applicable to various CNN architectures

Abstract

Deep neural networks typically impose significant computational loads and memory consumption. Moreover, the large parameters pose constraints on deploying the model on edge devices such as embedded systems. Tensor decomposition offers a clear advantage in compressing large-scale weight tensors. Nevertheless, direct utilization of low-rank decomposition typically leads to significant accuracy loss. This paper proposes a model compression method that integrates Variational Bayesian Matrix Factorization (VBMF) with orthogonal regularization. Initially, the model undergoes over-parameterization and training, with orthogonal regularization applied to enhance its likelihood of achieving the accuracy of the original model. Secondly, VBMF is employed to estimate the rank of the weight tensor at each layer. Our framework is sufficiently general to apply to other convolutional neural networks and easily adaptable to incorporate other tensor decomposition methods. Experimental results show that for both high and low compression ratios, our compression model exhibits advanced performance.