# Importance Estimation for Neural Network Pruning

## TL;DR

This paper introduces a new importance estimation method for neural network pruning using Taylor expansions, enabling effective layer-agnostic pruning that improves efficiency with minimal accuracy loss.

## Contribution

The paper presents a novel Taylor expansion-based importance estimation method for neural network pruning that scales across layers and types, outperforming previous techniques.

## Key findings

- High correlation (>93%) between importance scores and true importance.
- Achieved 40% FLOPS reduction on ResNet-101 with only 0.02% accuracy loss.
- Method outperforms state-of-the-art pruning techniques.

## Abstract

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and second-order Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-of-the-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning.