Dynamic Channel Pruning: Feature Boosting and Suppression

TL;DR

This paper introduces Feature Boosting and Suppression (FBS), a dynamic channel pruning method that adaptively amplifies important features and skips unimportant ones during runtime, improving efficiency without sacrificing accuracy.

Contribution

FBS is a novel dynamic channel pruning technique that preserves full network structure and accelerates computation by input-dependent feature suppression, trained with standard methods.

Findings

FBS achieves 5x compute savings on VGG-16 with less than 0.6% accuracy loss.

FBS achieves 2x compute savings on ResNet-18 with less than 0.6% accuracy loss.

FBS outperforms existing channel pruning and dynamic execution methods on ImageNet.

Abstract

Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources. In this paper, we reduce this cost by exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent, and propose feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time. FBS introduces small auxiliary connections to existing convolutional layers. In contrast to channel pruning methods which permanently remove channels, it preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels. FBS-augmented networks are trained with conventional stochastic gradient descent, making it readily available for many state-of-the-art CNNs. We compare FBS to a range of existing channel pruning and dynamic execution schemes and demonstrate large improvements on ImageNet classification. Experiments show that FBS can respectively provide $5\times$ and $2\times$ savings in compute on VGG-16 and ResNet-18, both with less than $0.6\%$ top-5 accuracy loss.