Pruning On-the-Fly: A Recoverable Pruning Method without Fine-tuning

TL;DR

This paper introduces a novel retraining-free pruning method for neural networks that uses hyperspherical learning and loss penalties, enabling instant recovery and minimal accuracy loss without fine-tuning.

Contribution

The proposed method allows for effective pruning without retraining and introduces a recovery technique by replacing pruned weights with their mean, outperforming existing approaches.

Findings

Achieves 50% pruning on ResNet-18 with less than 0.5% accuracy drop.

Significantly improves accuracy of pruned MobileNetV2 models compared to conventional methods.

Enables instant accuracy recovery by replacing pruned weights with their mean value.

Abstract

Most existing pruning works are resource-intensive, requiring retraining or fine-tuning of the pruned models for accuracy. We propose a retraining-free pruning method based on hyperspherical learning and loss penalty terms. The proposed loss penalty term pushes some of the model weights far from zero, while the rest weight values are pushed near zero and can be safely pruned with no need for retraining and a negligible accuracy drop. In addition, our proposed method can instantly recover the accuracy of a pruned model by replacing the pruned values with their mean value. Our method obtains state-of-the-art results in retraining-free pruning and is evaluated on ResNet-18/50 and MobileNetV2 with ImageNet dataset. One can easily get a 50\% pruned ResNet18 model with a 0.47\% accuracy drop. With fine-tuning, the experiment results show that our method can significantly boost the accuracy of the pruned models compared with existing works. For example, the accuracy of a 70\% pruned (except the first convolutional layer) MobileNetV2 model only drops 3.5\%, much less than the 7\% $\sim$ 10\% accuracy drop with conventional methods.