Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning
Seunghyun Lee, Byung Cheol Song

TL;DR
This paper introduces Ensemble Knowledge Guidance (EKG), a novel method for sub-network search and fine-tuning in filter pruning that effectively predicts potential performance and reduces fine-tuning costs.
Contribution
It proposes EKG to evaluate potential performance via loss landscape fluctuation and guides both search and fine-tuning with minimal additional cost.
Findings
EKG effectively predicts potential performance.
Pruning ResNet-50 with 45% FLOPS reduction in 315 GPU hours.
EKG-guided pruning maintains performance without degradation.
Abstract
Conventional NAS-based pruning algorithms aim to find the sub-network with the best validation performance. However, validation performance does not successfully represent test performance, i.e., potential performance. Also, although fine-tuning the pruned network to restore the performance drop is an inevitable process, few studies have handled this issue. This paper provides a novel Ensemble Knowledge Guidance (EKG) to solve both problems at once. First, we experimentally prove that the fluctuation of loss landscape can be an effective metric to evaluate the potential performance. In order to search a sub-network with the smoothest loss landscape at a low cost, we employ EKG as a search reward. EKG utilized for the following search iteration is composed of the ensemble knowledge of interim sub-networks, i.e., the by-products of the sub-network evaluation. Next, we reuse EKG to provide…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Parallel Computing and Optimization Techniques · Image and Signal Denoising Methods
MethodsPruning
