# Trained Rank Pruning for Efficient Deep Neural Networks

## TL;DR

This paper introduces Trained Rank Pruning (TRP), a novel method that integrates low-rank approximation with training to produce efficient neural networks with minimal performance loss, suitable for edge devices.

## Contribution

TRP combines low-rank approximation and training into a unified process, improving network compression without the need for fine-tuning after approximation.

## Key findings

- TRP outperforms previous low-rank approximation methods on CIFAR-10 and ImageNet.
- TRP maintains network capacity while enforcing low-rank structure.
- The method eliminates the need for post-approximation fine-tuning.

## Abstract

The performance of Deep Neural Networks (DNNs) keeps elevating in recent years with increasing network depth and width. To enable DNNs on edge devices like mobile phones, researchers proposed several network compression methods including pruning, quantization and factorization. Among the factorization-based approaches, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations. Several previous works attempted to directly approximate a pre-trained model by low-rank decomposition; however, small approximation errors in parameters can ripple a large prediction loss. As a result, performance usually drops significantly and a sophisticated fine-tuning is required to recover accuracy. We argue that it is not optimal to separate low-rank approximation from training. Unlike previous works, this paper integrates low rank approximation and regularization into the training. We propose Trained Rank Pruning (TRP), which iterates low rank approximation and training. TRP maintains the capacity of original network while imposes low-rank constraints during training. A stochastic sub-gradient descent optimized nuclear regularization is utilized to further encourage low rank in TRP. The TRP trained network has low-rank structure in nature, and can be approximated with negligible performance loss, eliminating fine-tuning after low rank approximation. The methods are comprehensively evaluated on CIFAR-10 and ImageNet, outperforming previous compression methods using low rank approximation. Code is available: https://github.com/yuhuixu1993/Trained-Rank-Pruning