Data-Driven Low-Rank Neural Network Compression

TL;DR

This paper introduces a data-driven low-rank approach to compress pretrained deep neural networks by reducing parameters in fully connected layers without retraining, maintaining accuracy and improving compression over existing methods.

Contribution

It proposes a novel convex optimization-based low-rank approximation method for DNN compression that outperforms sparsity-based techniques like Net-Trim.

Findings

Significantly reduces parameters with minimal accuracy loss

Outperforms Net-Trim in compression and accuracy retention

Applicable to common DNN architectures without retraining

Abstract

Despite many modern applications of Deep Neural Networks (DNNs), the large number of parameters in the hidden layers makes them unattractive for deployment on devices with storage capacity constraints. In this paper we propose a Data-Driven Low-rank (DDLR) method to reduce the number of parameters of pretrained DNNs and expedite inference by imposing low-rank structure on the fully connected layers, while controlling for the overall accuracy and without requiring any retraining. We pose the problem as finding the lowest rank approximation of each fully connected layer with given performance guarantees and relax it to a tractable convex optimization problem. We show that it is possible to significantly reduce the number of parameters in common DNN architectures with only a small reduction in classification accuracy. We compare DDLR with Net-Trim, which is another data-driven DNN compression technique based on sparsity and show that DDLR consistently produces more compressed neural networks while maintaining higher accuracy.