KCP: Kernel Cluster Pruning for Dense Labeling Neural Networks

TL;DR

This paper introduces Kernel Cluster Pruning (KCP), a novel method for pruning dense labeling neural networks by clustering kernels to better preserve network performance while significantly reducing FLOPs.

Contribution

KCP is a new pruning technique that clusters kernels to identify and remove the least representative ones, improving accuracy and efficiency in dense labeling tasks.

Findings

Reduces over 70% of FLOPs in dense labeling networks with less than 1% accuracy loss.

Achieves over 50% FLOPs reduction on ResNet-50 with a slight accuracy gain.

State-of-the-art pruning results for dense labeling neural networks.

Abstract

Pruning has become a promising technique used to compress and accelerate neural networks. Existing methods are mainly evaluated on spare labeling applications. However, dense labeling applications are those closer to real world problems that require real-time processing on resource-constrained mobile devices. Pruning for dense labeling applications is still a largely unexplored field. The prevailing filter channel pruning method removes the entire filter channel. Accordingly, the interaction between each kernel in one filter channel is ignored. In this study, we proposed kernel cluster pruning (KCP) to prune dense labeling networks. We developed a clustering technique to identify the least representational kernels in each layer. By iteratively removing those kernels, the parameter that can better represent the entire network is preserved; thus, we achieve better accuracy with a decent model size and computation reduction. When evaluated on stereo matching and semantic segmentation neural networks, our method can reduce more than 70% of FLOPs with less than 1% of accuracy drop. Moreover, for ResNet-50 on ILSVRC-2012, our KCP can reduce more than 50% of FLOPs reduction with 0.13% Top-1 accuracy gain. Therefore, KCP achieves state-of-the-art pruning results.