Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

TL;DR

This paper presents a data-driven neuron pruning method called network trimming that reduces neural network size by removing neurons with zero activations, maintaining or improving accuracy while decreasing computational costs.

Contribution

Introduces a novel iterative pruning algorithm based on neuron output analysis, enabling efficient network compression without sacrificing performance.

Findings

Achieves high compression ratios on LeNet and VGG-16.

Maintains or improves accuracy after pruning.

Reduces computational and memory costs significantly.

Abstract

State-of-the-art neural networks are getting deeper and wider. While their performance increases with the increasing number of layers and neurons, it is crucial to design an efficient deep architecture in order to reduce computational and memory costs. Designing an efficient neural network, however, is labor intensive requiring many experiments, and fine-tunings. In this paper, we introduce network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset. Our algorithm is inspired by an observation that the outputs of a significant portion of neurons in a large network are mostly zero, regardless of what inputs the network received. These zero activation neurons are redundant, and can be removed without affecting the overall accuracy of the network. After pruning the zero activation neurons, we retrain the network using the weights before pruning as initialization. We alternate the pruning and retraining to further reduce zero activations in a network. Our experiments on the LeNet and VGG-16 show that we can achieve high compression ratio of parameters without losing or even achieving higher accuracy than the original network.