End-to-End Sensitivity-Based Filter Pruning

TL;DR

This paper introduces SbF-Pruner, a sensitivity-based filter pruning method that learns filter importance scores end-to-end, enabling efficient, one-stage training of pruned networks without pretraining or layer-specific hyperparameters.

Contribution

The novel SbF-Pruner algorithm jointly learns filter importance scores from scratch, accounting for layer interdependencies and outperforming existing pruning methods in accuracy and parameter reduction.

Findings

Achieves 1.02% and 1.19% accuracy gains on CIFAR-10 for ResNet56 and ResNet110.

Reduces parameters by over 52% while outperforming state-of-the-art pruning methods.

Operates without pretraining or layer-specific hyperparameters.

Abstract

In this paper, we present a novel sensitivity-based filter pruning algorithm (SbF-Pruner) to learn the importance scores of filters of each layer end-to-end. Our method learns the scores from the filter weights, enabling it to account for the correlations between the filters of each layer. Moreover, by training the pruning scores of all layers simultaneously our method can account for layer interdependencies, which is essential to find a performant sparse sub-network. Our proposed method can train and generate a pruned network from scratch in a straightforward, one-stage training process without requiring a pretrained network. Ultimately, we do not need layer-specific hyperparameters and pre-defined layer budgets, since SbF-Pruner can implicitly determine the appropriate number of channels in each layer. Our experimental results on different network architectures suggest that SbF-Pruner outperforms advanced pruning methods. Notably, on CIFAR-10, without requiring a pretrained baseline network, we obtain 1.02% and 1.19% accuracy gain on ResNet56 and ResNet110, compared to the baseline reported for state-of-the-art pruning algorithms. This is while SbF-Pruner reduces parameter-count by 52.3% (for ResNet56) and 54% (for ResNet101), which is better than the state-of-the-art pruning algorithms with a high margin of 9.5% and 6.6%.