CWP: Instance complexity weighted channel-wise soft masks for network pruning

TL;DR

This paper introduces CWP, a differentiable network pruning method that incorporates instance complexity to weight filter importance, effectively preserving model accuracy while significantly reducing FLOPs.

Contribution

It proposes a novel instance complexity weighted importance scoring and a regularizer for mask polarization, improving pruning effectiveness over existing methods.

Findings

CWP improves ResNet56 accuracy by 0.32% on CIFAR-10 after 64.11% FLOPs reduction.

CWP prunes 87.75% FLOPs of ResNet50 on ImageNet with only 0.93% accuracy loss.

Demonstrates advantages over state-of-the-art pruning techniques across various architectures and datasets.

Abstract

Existing differentiable channel pruning methods often attach scaling factors or masks behind channels to prune filters with less importance, and implicitly assume uniform contribution of input samples to filter importance. Specifically, the effects of instance complexity on pruning performance are not yet fully investigated in static network pruning. In this paper, we propose a simple yet effective differentiable network pruning method CWP based on instance complexity weighted filter importance scores. We define instance complexity related weight for each instance by giving higher weights to hard instances, and measure the weighted sum of instance-specific soft masks to model non-uniform contribution of different inputs, which encourages hard instances to dominate the pruning process and the model performance to be well preserved. In addition, we introduce a regularizer to maximize polarization of the masks, such that a sweet spot can be easily found to identify the filters to be pruned. Performance evaluations on various network architectures and datasets demonstrate CWP has advantages over the state-of-the-arts in pruning large networks. For instance, CWP improves the accuracy of ResNet56 on CIFAR-10 dataset by 0.32% aftering removing 64.11% FLOPs, and prunes 87.75% FLOPs of ResNet50 on ImageNet dataset with only 0.93% Top-1 accuracy loss.