Importance Estimation with Random Gradient for Neural Network Pruning

TL;DR

This paper introduces a novel importance estimation technique for neural network pruning that uses random gradients and normalization to improve efficiency without requiring labeled data, outperforming existing methods.

Contribution

It proposes a new importance estimation method using random gradients and normalization, enhancing pruning performance and complementing existing approaches.

Findings

Outperforms previous importance estimation methods on ResNet and VGG.

Effective without labeled examples due to random gradient propagation.

Improves existing methods when combined with them.

Abstract

Global Neuron Importance Estimation is used to prune neural networks for efficiency reasons. To determine the global importance of each neuron or convolutional kernel, most of the existing methods either use activation or gradient information or both, which demands abundant labelled examples. In this work, we use heuristics to derive importance estimation similar to Taylor First Order (TaylorFO) approximation based methods. We name our methods TaylorFO-abs and TaylorFO-sq. We propose two additional methods to improve these importance estimation methods. Firstly, we propagate random gradients from the last layer of a network, thus avoiding the need for labelled examples. Secondly, we normalize the gradient magnitude of the last layer output before propagating, which allows all examples to contribute similarly to the importance score. Our methods with additional techniques perform better than previous methods when tested on ResNet and VGG architectures on CIFAR-100 and STL-10 datasets. Furthermore, our method also complements the existing methods and improves their performances when combined with them.