Receding Neuron Importances for Structured Pruning

TL;DR

This paper introduces a novel structured pruning method that uses a bounded BatchNorm variation and a regularisation term to effectively identify and remove unimportant neurons, enabling larger and less damaging network compression.

Contribution

The paper proposes a new regularisation technique with bounded BatchNorm parameters that selectively suppresses low-importance neurons, improving pruning efficiency and performance.

Findings

Enables larger pruning ratios with less accuracy loss.

Outperforms existing pruning methods on VGG and ResNet architectures.

Achieves effective one-shot pruning on CIFAR and ImageNet datasets.

Abstract

Structured pruning efficiently compresses networks by identifying and removing unimportant neurons. While this can be elegantly achieved by applying sparsity-inducing regularisation on BatchNorm parameters, an L1 penalty would shrink all scaling factors rather than just those of superfluous neurons. To tackle this issue, we introduce a simple BatchNorm variation with bounded scaling parameters, based on which we design a novel regularisation term that suppresses only neurons with low importance. Under our method, the weights of unnecessary neurons effectively recede, producing a polarised bimodal distribution of importances. We show that neural networks trained this way can be pruned to a larger extent and with less deterioration. We one-shot prune VGG and ResNet architectures at different ratios on CIFAR and ImagenNet datasets. In the case of VGG-style networks, our method significantly outperforms existing approaches particularly under a severe pruning regime.