Model Compression using Progressive Channel Pruning

TL;DR

This paper introduces Progressive Channel Pruning (PCP), a novel iterative framework for CNN channel pruning that improves efficiency and accuracy retention, applicable to both supervised and transfer learning scenarios.

Contribution

The paper presents a new progressive pruning framework that iteratively prunes channels based on estimated accuracy drops, outperforming existing methods in supervised and transfer learning.

Findings

Outperforms existing channel pruning methods in benchmarks.

Effectively reduces data distribution mismatch in transfer learning.

Maintains higher accuracy with fewer channels pruned.

Abstract

In this work, we propose a simple but effective channel pruning framework called Progressive Channel Pruning (PCP) to accelerate Convolutional Neural Networks (CNNs). In contrast to the existing channel pruning methods that prune channels only once per layer in a layer-by-layer fashion, our new progressive framework iteratively prunes a small number of channels from several selected layers, which consists of a three-step attempting-selecting-pruning pipeline in each iteration. In the attempting step, we attempt to prune a pre-defined number of channels from one layer by using any existing channel pruning methods and estimate the accuracy drop for this layer based on the labelled samples in the validation set. In the selecting step, based on the estimated accuracy drops for all layers, we propose a greedy strategy to automatically select a set of layers that will lead to less overall accuracy drop after pruning these layers. In the pruning step, we prune a small number of channels from these selected layers. We further extend our PCP framework to prune channels for the deep transfer learning methods like Domain Adversarial Neural Network (DANN), in which we effectively reduce the data distribution mismatch in the channel pruning process by using both labelled samples from the source domain and pseudo-labelled samples from the target domain. Our comprehensive experiments on two benchmark datasets demonstrate that our PCP framework outperforms the existing channel pruning approaches under both supervised learning and transfer learning settings.