Effective Model Compression via Stage-wise Pruning

TL;DR

This paper introduces a stage-wise pruning method for deep CNNs that improves supernet training fairness and completeness, leading to more effective model compression and better performance on benchmark datasets.

Contribution

The proposed stage-wise pruning (SWP) method addresses unfull and unfair training issues in Auto-ML pruning by splitting supernets and using inplace distillation, achieving state-of-the-art results.

Findings

SWP improves proxy performance accuracy.

SWP outperforms previous Auto-ML pruning methods.

Achieves state-of-the-art on CIFAR-10 and ImageNet.

Abstract

Automated Machine Learning(Auto-ML) pruning methods aim at searching a pruning strategy automatically to reduce the computational complexity of deep Convolutional Neural Networks(deep CNNs). However, some previous work found that the results of many Auto-ML pruning methods cannot even surpass the results of the uniformly pruning method. In this paper, the ineffectiveness of Auto-ML pruning which is caused by unfull and unfair training of the supernet is shown. A deep supernet suffers from unfull training because it contains too many candidates. To overcome the unfull training, a stage-wise pruning(SWP) method is proposed, which splits a deep supernet into several stage-wise supernets to reduce the candidate number and utilize inplace distillation to supervise the stage training. Besides, A wide supernet is hit by unfair training since the sampling probability of each channel is unequal. Therefore, the fullnet and the tinynet are sampled in each training iteration to ensure each channel can be overtrained. Remarkably, the proxy performance of the subnets trained with SWP is closer to the actual performance than that of most of the previous Auto-ML pruning work. Experiments show that SWP achieves the state-of-the-art on both CIFAR-10 and ImageNet under the mobile setting.