Filter Sketch for Network Pruning
Mingbao Lin, Liujuan Cao, Shaojie Li, Qixiang Ye, Yonghong Tian,, Jianzhuang Liu, Qi Tian, Rongrong Ji

TL;DR
FilterSketch is a network pruning method that preserves information using matrix sketching, significantly reducing computational cost while maintaining accuracy, demonstrated on CIFAR-10 and ImageNet with ResNet models.
Contribution
The paper introduces FilterSketch, a novel pruning approach that encodes second-order information via matrix sketching, avoiding retraining and data-driven optimization.
Findings
Reduces 63.3% FLOPs on CIFAR-10 with negligible accuracy loss.
Prunes 43% parameters on ImageNet with only 0.69% accuracy drop.
Requires no training from scratch or iterative data-driven optimization.
Abstract
We propose a novel network pruning approach by information preserving of pre-trained network weights (filters). Network pruning with the information preserving is formulated as a matrix sketch problem, which is efficiently solved by the off-the-shelf Frequent Direction method. Our approach, referred to as FilterSketch, encodes the second-order information of pre-trained weights, which enables the representation capacity of pruned networks to be recovered with a simple fine-tuning procedure. FilterSketch requires neither training from scratch nor data-driven iterative optimization, leading to a several-orders-of-magnitude reduction of time cost in the optimization of pruning. Experiments on CIFAR-10 show that FilterSketch reduces 63.3% of FLOPs and prunes 59.9% of network parameters with negligible accuracy cost for ResNet-110. On ILSVRC-2012, it reduces 45.5% of FLOPs and removes 43.0%…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Underwater Acoustics Research
MethodsPruning
