Towards thinner convolutional neural networks through Gradually Global Pruning

TL;DR

This paper introduces a gradually global neuron pruning method that efficiently reduces deep neural network size by removing redundant neurons across all layers, maintaining performance while creating thinner models.

Contribution

It proposes a novel global pruning scheme with bias elimination for neuron importance, improving over layer-wise methods for deep networks.

Findings

Automatically finds thinner sub-networks with maintained performance.

More effective than layer-wise pruning for deep neural networks.

Reduces model size and computation cost.

Abstract

Deep network pruning is an effective method to reduce the storage and computation cost of deep neural networks when applying them to resource-limited devices. Among many pruning granularities, neuron level pruning will remove redundant neurons and filters in the model and result in thinner networks. In this paper, we propose a gradually global pruning scheme for neuron level pruning. In each pruning step, a small percent of neurons were selected and dropped across all layers in the model. We also propose a simple method to eliminate the biases in evaluating the importance of neurons to make the scheme feasible. Compared with layer-wise pruning scheme, our scheme avoid the difficulty in determining the redundancy in each layer and is more effective for deep networks. Our scheme would automatically find a thinner sub-network in original network under a given performance.