SNIP: Single-shot Network Pruning based on Connection Sensitivity
Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr

TL;DR
SNIP introduces a one-shot pruning method based on connection sensitivity at initialization, enabling the creation of highly sparse neural networks without iterative pruning or hyperparameter tuning, while maintaining accuracy.
Contribution
The paper presents a novel single-shot pruning technique that prunes networks at initialization using connection sensitivity, eliminating the need for training-based pruning schedules.
Findings
Achieves extremely sparse networks with similar accuracy to dense models.
Applicable to various architectures including CNNs, residual, and recurrent networks.
Effective across multiple datasets like MNIST, CIFAR-10, and Tiny-ImageNet.
Abstract
Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Adversarial Robustness in Machine Learning
MethodsPruning
