A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks

TL;DR

This paper introduces a Layer Decomposition-Recomposition Framework (LDRF) for neuron pruning that preserves information during network compression, leading to more accurate lightweight models with minimal accuracy loss.

Contribution

The novel LDRF method propagates complete layer information in an embedding space, outperforming traditional layer-by-layer pruning approaches in accuracy and efficiency.

Findings

Achieves 5.13x and 3x speed-up on VGG-16 and ResNet-50 with minimal accuracy loss.

Significantly better results before end-to-end fine-tuning compared to existing methods.

Outperforms state-of-the-art neuron pruning techniques on ILSVRC-12 benchmark.

Abstract

Neuron pruning is an efficient method to compress the network into a slimmer one for reducing the computational cost and storage overhead. Most of state-of-the-art results are obtained in a layer-by-layer optimization mode. It discards the unimportant input neurons and uses the survived ones to reconstruct the output neurons approaching to the original ones in a layer-by-layer manner. However, an unnoticed problem arises that the information loss is accumulated as layer increases since the survived neurons still do not encode the entire information as before. A better alternative is to propagate the entire useful information to reconstruct the pruned layer instead of directly discarding the less important neurons. To this end, we propose a novel Layer Decomposition-Recomposition Framework (LDRF) for neuron pruning, by which each layer's output information is recovered in an embedding space and then propagated to reconstruct the following pruned layers with useful information preserved. We mainly conduct our experiments on ILSVRC-12 benchmark with VGG-16 and ResNet-50. What should be emphasized is that our results before end-to-end fine-tuning are significantly superior owing to the information-preserving property of our proposed framework.With end-to-end fine-tuning, we achieve state-of-the-art results of 5.13x and 3x speed-up with only 0.5% and 0.65% top-5 accuracy drop respectively, which outperform the existing neuron pruning methods.