Pruning Networks with Cross-Layer Ranking & k-Reciprocal Nearest Filters

TL;DR

This paper introduces CLR-RNF, a novel filter pruning method that effectively reduces computational cost and parameters in neural networks by using cross-layer ranking and reciprocal nearest filter selection, outperforming existing methods.

Contribution

The paper proposes a non-learning, computation-aware filter pruning approach combining cross-layer ranking and reciprocal nearest filter selection, addressing the long-tail pruning problem and reducing complexity.

Findings

Significant FLOPs and parameter reduction on CIFAR-10 and ImageNet.

Achieves accuracy improvements or minimal drops with high pruning ratios.

Outperforms state-of-the-art pruning methods in experiments.

Abstract

This paper focuses on filter-level network pruning. A novel pruning method, termed CLR-RNF, is proposed. We first reveal a "long-tail" long-tail pruning problem in magnitude-based weight pruning methods, and then propose a computation-aware measurement for individual weight importance, followed by a Cross-Layer Ranking (CLR) of weights to identify and remove the bottom-ranked weights. Consequently, the per-layer sparsity makes up of the pruned network structure in our filter pruning. Then, we introduce a recommendation-based filter selection scheme where each filter recommends a group of its closest filters. To pick the preserved filters from these recommended groups, we further devise a k-Reciprocal Nearest Filter (RNF) selection scheme where the selected filters fall into the intersection of these recommended groups. Both our pruned network structure and the filter selection are non-learning processes, which thus significantly reduce the pruning complexity, and differentiate our method from existing works. We conduct image classification on CIFAR-10 and ImageNet to demonstrate the superiority of our CLR-RNF over the state-of-the-arts. For example, on CIFAR-10, CLR-RNF removes 74.1% FLOPs and 95.0% parameters from VGGNet-16 with even 0.3\% accuracy improvements. On ImageNet, it removes 70.2% FLOPs and 64.8% parameters from ResNet-50 with only 1.7% top-5 accuracy drops. Our project is at https://github.com/lmbxmu/CLR-RNF.