Holistic Filter Pruning for Efficient Deep Neural Networks

TL;DR

Holistic Filter Pruning (HFP) is a new method for efficiently reducing the complexity of deep neural networks by globally allocating pruning resources to meet specific size targets, achieving state-of-the-art results.

Contribution

HFP introduces a simple, global pruning approach that allows precise control over model size and complexity during training, outperforming existing methods.

Findings

Prunes 60% of multiplications in ResNet-50 on ImageNet with minimal accuracy loss.

Achieves state-of-the-art pruning performance on CIFAR-10 and ImageNet.

Enables easy implementation and accurate control of pruning rates.

Abstract

Deep neural networks (DNNs) are usually over-parameterized to increase the likelihood of getting adequate initial weights by random initialization. Consequently, trained DNNs have many redundancies which can be pruned from the model to reduce complexity and improve the ability to generalize. Structural sparsity, as achieved by filter pruning, directly reduces the tensor sizes of weights and activations and is thus particularly effective for reducing complexity. We propose "Holistic Filter Pruning" (HFP), a novel approach for common DNN training that is easy to implement and enables to specify accurate pruning rates for the number of both parameters and multiplications. After each forward pass, the current model complexity is calculated and compared to the desired target size. By gradient descent, a global solution can be found that allocates the pruning budget over the individual layers such that the desired target size is fulfilled. In various experiments, we give insights into the training and achieve state-of-the-art performance on CIFAR-10 and ImageNet (HFP prunes 60% of the multiplications of ResNet-50 on ImageNet with no significant loss in the accuracy). We believe our simple and powerful pruning approach to constitute a valuable contribution for users of DNNs in low-cost applications.