Dependency Aware Filter Pruning

TL;DR

This paper introduces a dependency-aware filter pruning method for CNNs that considers layer dependencies and dynamically controls sparsity, leading to more efficient network compression.

Contribution

It develops a norm-based importance estimation that accounts for layer dependencies and proposes a dynamic regularization mechanism for better filter pruning.

Findings

Outperforms existing methods on CIFAR, SVHN, and ImageNet datasets.

Effectively identifies unimportant filters considering layer dependencies.

Achieves desired sparsity with improved accuracy and efficiency.

Abstract

Convolutional neural networks (CNNs) are typically over-parameterized, bringing considerable computational overhead and memory footprint in inference. Pruning a proportion of unimportant filters is an efficient way to mitigate the inference cost. For this purpose, identifying unimportant convolutional filters is the key to effective filter pruning. Previous work prunes filters according to either their weight norms or the corresponding batch-norm scaling factors, while neglecting the sequential dependency between adjacent layers. In this paper, we further develop the norm-based importance estimation by taking the dependency between the adjacent layers into consideration. Besides, we propose a novel mechanism to dynamically control the sparsity-inducing regularization so as to achieve the desired sparsity. In this way, we can identify unimportant filters and search for the optimal network architecture within certain resource budgets in a more principled manner. Comprehensive experimental results demonstrate the proposed method performs favorably against the existing strong baseline on the CIFAR, SVHN, and ImageNet datasets. The training sources will be publicly available after the review process.