Low-Rank+Sparse Tensor Compression for Neural Networks

TL;DR

This paper introduces a novel tensor compression method combining low-rank decomposition with sparse pruning to effectively reduce neural network size, especially for modern architectures like MobileNet and EfficientNet.

Contribution

It proposes a combined low-rank and sparse tensor compression technique that leverages both coarse and fine network structures for improved neural network compression.

Findings

Outperforms pure tensor decomposition and sparse pruning individually.

Effective on state-of-the-art architectures like MobileNetv3, EfficientNet, and Vision Transformer.

Reduces memory and computation requirements while maintaining accuracy.

Abstract

Low-rank tensor compression has been proposed as a promising approach to reduce the memory and compute requirements of neural networks for their deployment on edge devices. Tensor compression reduces the number of parameters required to represent a neural network weight by assuming network weights possess a coarse higher-order structure. This coarse structure assumption has been applied to compress large neural networks such as VGG and ResNet. However modern state-of-the-art neural networks for computer vision tasks (i.e. MobileNet, EfficientNet) already assume a coarse factorized structure through depthwise separable convolutions, making pure tensor decomposition a less attractive approach. We propose to combine low-rank tensor decomposition with sparse pruning in order to take advantage of both coarse and fine structure for compression. We compress weights in SOTA architectures (MobileNetv3, EfficientNet, Vision Transformer) and compare this approach to sparse pruning and tensor decomposition alone.