WeightMom: Learning Sparse Networks using Iterative Momentum-based pruning

TL;DR

WeightMom introduces an iterative momentum-based pruning method that effectively reduces neural network size while maintaining accuracy, outperforming previous approaches on popular image classification models.

Contribution

The paper presents a novel weight pruning technique using momentum to guide gradual sparsification, improving compression and accuracy retention.

Findings

Achieved 15% compression with minimal accuracy loss.

Outperformed previous pruning methods on AlexNet, VGG16, ResNet50.

Effective on CIFAR-10 and CIFAR-100 datasets.

Abstract

Deep Neural Networks have been used in a wide variety of applications with significant success. However, their highly complex nature owing to comprising millions of parameters has lead to problems during deployment in pipelines with low latency requirements. As a result, it is more desirable to obtain lightweight neural networks which have the same performance during inference time. In this work, we propose a weight based pruning approach in which the weights are pruned gradually based on their momentum of the previous iterations. Each layer of the neural network is assigned an importance value based on their relative sparsity, followed by the magnitude of the weight in the previous iterations. We evaluate our approach on networks such as AlexNet, VGG16 and ResNet50 with image classification datasets such as CIFAR-10 and CIFAR-100. We found that the results outperformed the previous approaches with respect to accuracy and compression ratio. Our method is able to obtain a compression of 15% for the same degradation in accuracy on both the datasets.