Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning

TL;DR

This paper introduces a dynamic rank pruning method for CNN compression that uses SVD and regularization to reduce parameters efficiently while preserving or improving accuracy across multiple datasets.

Contribution

It presents a novel end-to-end training framework combining matrix factorization and regularization for dynamic low-rank CNN compression.

Findings

Significant model size reduction with maintained or improved accuracy.

Effective application across various CNN architectures and datasets.

Enhanced training efficiency through end-to-end SVD factor optimization.

Abstract

While Convolutional Neural Networks (CNNs) excel at learning complex latent-space representations, their over-parameterization can lead to overfitting and reduced performance, particularly with limited data. This, alongside their high computational and memory demands, limits the applicability of CNNs for edge deployment. Low-rank matrix approximation has emerged as a promising approach to reduce CNN parameters, but its application presents challenges including rank selection and performance loss. To address these issues, we propose an efficient training method for CNN compression via dynamic parameter rank pruning. Our approach integrates efficient matrix factorization and novel regularization techniques, forming a robust framework for dynamic rank reduction and model compression. We use Singular Value Decomposition (SVD) to model low-rank convolutional filters and dense weight matrices and we achieve model compression by training the SVD factors with back-propagation in an end-to-end way. We evaluate our method on an array of modern CNNs, including ResNet-18, ResNet-20, and ResNet-32, and datasets like CIFAR-10, CIFAR-100, and ImageNet (2012), showcasing its applicability in computer vision. Our experiments show that the proposed method can yield substantial storage savings while maintaining or even enhancing classification performance.