Neural Network Compression via Effective Filter Analysis and Hierarchical Pruning

TL;DR

This paper introduces a theoretically grounded hierarchical pruning method for neural network compression, effectively estimating maximum redundancy and preventing over-pruning, leading to superior performance on CNNs.

Contribution

It presents a novel gradient-matrix singularity analysis framework to estimate maximum network redundancy and a hierarchical pruning algorithm that preserves performance.

Findings

Achieved state-of-the-art pruning performance.

Maintained high accuracy at high compression ratios.

Validated on multiple CNN architectures.

Abstract

Network compression is crucial to making the deep networks to be more efficient, faster, and generalizable to low-end hardware. Current network compression methods have two open problems: first, there lacks a theoretical framework to estimate the maximum compression rate; second, some layers may get over-prunned, resulting in significant network performance drop. To solve these two problems, this study propose a gradient-matrix singularity analysis-based method to estimate the maximum network redundancy. Guided by that maximum rate, a novel and efficient hierarchical network pruning algorithm is developed to maximally condense the neuronal network structure without sacrificing network performance. Substantial experiments are performed to demonstrate the efficacy of the new method for pruning several advanced convolutional neural network (CNN) architectures. Compared to existing pruning methods, the proposed pruning algorithm achieved state-of-the-art performance. At the same or similar compression ratio, the new method provided the highest network prediction accuracy as compared to other methods.