Dynamic Model Pruning with Feedback

TL;DR

This paper introduces a dynamic model pruning technique that creates sparse neural networks with feedback mechanisms, achieving state-of-the-art performance without retraining, suitable for deployment on low-end devices.

Contribution

The method enables dynamic sparsity pattern allocation and feedback-driven reactivation of pruned weights, improving efficiency and performance in a single training pass.

Findings

Achieves state-of-the-art accuracy on CIFAR-10 and ImageNet with sparse models.

Outperforms previous pruning schemes in model performance.

No retraining needed for the sparse models.

Abstract

Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models. Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.