GHFP: Gradually Hard Filter Pruning

TL;DR

GHFP introduces a gradual transition from soft to hard filter pruning during training, combining the advantages of both methods to improve performance and convergence speed in deep neural network pruning.

Contribution

The paper proposes a novel Gradually Hard Filter Pruning (GHFP) method that smoothly switches from SFP to HFP, enhancing pruning efficiency and model performance.

Findings

Achieves state-of-the-art results on CIFAR-10/100.

Balances pruning speed and accuracy effectively.

Maintains larger search space initially, then reduces capacity gradually.

Abstract

Filter pruning is widely used to reduce the computation of deep learning, enabling the deployment of Deep Neural Networks (DNNs) in resource-limited devices. Conventional Hard Filter Pruning (HFP) method zeroizes pruned filters and stops updating them, thus reducing the search space of the model. On the contrary, Soft Filter Pruning (SFP) simply zeroizes pruned filters, keeping updating them in the following training epochs, thus maintaining the capacity of the network. However, SFP, together with its variants, converges much slower than HFP due to its larger search space. Our question is whether SFP-based methods and HFP can be combined to achieve better performance and speed up convergence. Firstly, we generalize SFP-based methods and HFP to analyze their characteristics. Then we propose a Gradually Hard Filter Pruning (GHFP) method to smoothly switch from SFP-based methods to HFP during training and pruning, thus maintaining a large search space at first, gradually reducing the capacity of the model to ensure a moderate convergence speed. Experimental results on CIFAR-10/100 show that our method achieves the state-of-the-art performance.