Exploring Gradient Flow Based Saliency for DNN Model Compression

TL;DR

This paper introduces a novel gradient flow-based saliency method for DNN model pruning, which considers the influence of batch normalization and ReLU layers, leading to improved model compression performance.

Contribution

It proposes a new channel importance evaluation method based on gradient flow and Taylor expansion, integrating BN and ReLU effects for more accurate pruning.

Findings

Outperforms traditional pruning methods in experiments

Effective on image classification and denoising tasks

Provides a theoretical basis for gradient-based channel importance

Abstract

Model pruning aims to reduce the deep neural network (DNN) model size or computational overhead. Traditional model pruning methods such as l-1 pruning that evaluates the channel significance for DNN pay too much attention to the local analysis of each channel and make use of the magnitude of the entire feature while ignoring its relevance to the batch normalization (BN) and ReLU layer after each convolutional operation. To overcome these problems, we propose a new model pruning method from a new perspective of gradient flow in this paper. Specifically, we first theoretically analyze the channel's influence based on Taylor expansion by integrating the effects of BN layer and ReLU activation function. Then, the incorporation of the first-order Talyor polynomial of the scaling parameter and the shifting parameter in the BN layer is suggested to effectively indicate the significance of a channel in a DNN. Comprehensive experiments on both image classification and image denoising tasks demonstrate the superiority of the proposed novel theory and scheme. Code is available at https://github.com/CityU-AIM-Group/GFBS.