Neural Network Compression by Joint Sparsity Promotion and Redundancy Reduction

TL;DR

This paper introduces a novel neural network compression method combining filter pruning and a gradient counting-based sparsity scheme, significantly reducing model size and memory without performance loss, and outperforming existing methods on segmentation benchmarks.

Contribution

A new training scheme that jointly promotes sparsity and prunes redundant filters using gradient counting, improving compression and performance.

Findings

Significant reduction in network parameters and memory footprint.

Improved performance over MobileNetV3 on segmentation tasks.

Effective compression without accuracy degradation.

Abstract

Compression of convolutional neural network models has recently been dominated by pruning approaches. A class of previous works focuses solely on pruning the unimportant filters to achieve network compression. Another important direction is the design of sparsity-inducing constraints which has also been explored in isolation. This paper presents a novel training scheme based on composite constraints that prune redundant filters and minimize their effect on overall network learning via sparsity promotion. Also, as opposed to prior works that employ pseudo-norm-based sparsity-inducing constraints, we propose a sparse scheme based on gradient counting in our framework. Our tests on several pixel-wise segmentation benchmarks show that the number of neurons and the memory footprint of networks in the test phase are significantly reduced without affecting performance. MobileNetV3 and UNet, two well-known architectures, are used to test the proposed scheme. Our network compression method not only results in reduced parameters but also achieves improved performance compared to MobileNetv3, which is an already optimized architecture.