Structured Pruning of Deep Convolutional Neural Networks

TL;DR

This paper proposes a structured pruning method for deep convolutional neural networks that enhances computational efficiency and reduces storage by introducing various scales of sparsity, optimized through particle filtering.

Contribution

It introduces a novel structured sparsity approach at multiple scales and employs particle filtering to identify important network connections, improving efficiency and hardware compatibility.

Findings

Significant reduction in network size and storage requirements.

Enhanced suitability for embedded and hardware systems.

Effective pruning with minimal accuracy loss.

Abstract

Real time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also do not fit well on parallel computation. We introduce structured sparsity at various scales for convolutional neural networks, which are channel wise, kernel wise and intra kernel strided sparsity. This structured sparsity is very advantageous for direct computational resource savings on embedded computers, parallel computing environments and hardware based systems. To decide the importance of network connections and paths, the proposed method uses a particle filtering approach. The importance weight of each particle is assigned by computing the misclassification rate with corresponding connectivity pattern. The pruned network is re-trained to compensate for the losses due to pruning. While implementing convolutions as matrix products, we particularly show that intra kernel strided sparsity with a simple constraint can significantly reduce the size of kernel and feature map matrices. The pruned network is finally fixed point optimized with reduced word length precision. This results in significant reduction in the total storage size providing advantages for on-chip memory based implementations of deep neural networks.