Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition

TL;DR

This paper introduces LoRITa, a training method that promotes low-rank structures in neural networks through layer composition and singular value truncation, enabling effective compression without structural changes at inference.

Contribution

LoRITa is a novel low-rank training technique that does not require pre-trained models, rank pre-specification, or iterative SVD, simplifying network compression.

Findings

Achieves competitive or state-of-the-art compression results on multiple datasets.

Does not require pre-training or rank pre-specification.

Effective across various neural network architectures.

Abstract

Deep Neural Networks (DNNs) have achieved remarkable success in addressing many previously unsolvable tasks. However, the storage and computational requirements associated with DNNs pose a challenge for deploying these trained models on resource-limited devices. Therefore, a plethora of compression and pruning techniques have been proposed in recent years. Low-rank decomposition techniques are among the approaches most utilized to address this problem. Compared to post-training compression, compression-promoted training is still under-explored. In this paper, we present a theoretically-justified technique termed Low-Rank Induced Training (LoRITa), that promotes low-rankness through the composition of linear layers and compresses by using singular value truncation. This is achieved without the need to change the structure at inference time or require constrained and/or additional optimization, other than the standard weight decay regularization. Moreover, LoRITa eliminates the need to (i) initialize with pre-trained models, (ii) specify rank selection prior to training, and (iii) compute SVD in each iteration. Our experimental results (i) demonstrate the effectiveness of our approach using MNIST on Fully Connected Networks, CIFAR10 on Vision Transformers, and CIFAR10/100 and ImageNet on Convolutional Neural Networks, and (ii) illustrate that we achieve either competitive or state-of-the-art results when compared to leading structured pruning and low-rank training methods in terms of FLOPs and parameters drop. Our code is available at \url{https://github.com/XitongSystem/LoRITa/tree/main}.