On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections

TL;DR

This paper introduces a novel, model-free structured pruning method for deep neural networks with skip connections, utilizing energy statistics to measure and prune redundant layers based on their dependency with outputs.

Contribution

The paper presents a new dependence measure based on energy statistics for pruning DNNs without parametric assumptions, suitable for high-dimensional feature spaces.

Findings

Effective pruning of skip-connection networks demonstrated

Competitive performance with state-of-the-art methods

Applicable to various neural network architectures

Abstract

We propose a new structured pruning framework for compressing Deep Neural Networks (DNNs) with skip connections, based on measuring the statistical dependency of hidden layers and predicted outputs. The dependence measure defined by the energy statistics of hidden layers serves as a model-free measure of information between the feature maps and the output of the network. The estimated dependence measure is subsequently used to prune a collection of redundant and uninformative layers. Model-freeness of our measure guarantees that no parametric assumptions on the feature map distribution are required, making it computationally appealing for very high dimensional feature space in DNNs. Extensive numerical experiments on various architectures show the efficacy of the proposed pruning approach with competitive performance to state-of-the-art methods.