SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance

TL;DR

SInGE introduces a novel neuron relevance criterion based on integrated gradients for structured pruning, leading to more efficient neural networks with minimal accuracy loss, validated across multiple datasets and architectures.

Contribution

The paper proposes a new integrated gradient-based pruning method, SInGE, that improves over existing heuristics and includes a combined pruning and fine-tuning process.

Findings

SInGE outperforms state-of-the-art pruning methods in accuracy and efficiency.

The integrated gradient criterion effectively identifies neuron importance.

The combined pruning and fine-tuning flow preserves model performance across datasets.

Abstract

The leap in performance in state-of-the-art computer vision methods is attributed to the development of deep neural networks. However it often comes at a computational price which may hinder their deployment. To alleviate this limitation, structured pruning is a well known technique which consists in removing channels, neurons or filters, and is commonly applied in order to produce more compact models. In most cases, the computations to remove are selected based on a relative importance criterion. At the same time, the need for explainable predictive models has risen tremendously and motivated the development of robust attribution methods that highlight the relative importance of pixels of an input image or feature map. In this work, we discuss the limitations of existing pruning heuristics, among which magnitude and gradient-based methods. We draw inspiration from attribution methods to design a novel integrated gradient pruning criterion, in which the relevance of each neuron is defined as the integral of the gradient variation on a path towards this neuron removal. Furthermore, we propose an entwined DNN pruning and fine-tuning flowchart to better preserve DNN accuracy while removing parameters. We show through extensive validation on several datasets, architectures as well as pruning scenarios that the proposed method, dubbed SInGE, significantly outperforms existing state-of-the-art DNN pruning methods.