On the Compression of Neural Networks Using $\ell_0$-Norm Regularization and Weight Pruning

TL;DR

This paper introduces a novel neural network compression method combining $\,\ell_0$-norm regularization for sparsity, weight pruning, and fine-tuning, effectively reducing model size while maintaining accuracy, suitable for edge and embedded applications.

Contribution

The paper develops a new $\,\ell_0$-norm regularization technique for inducing sparsity and integrates it with pruning and fine-tuning for improved neural network compression.

Findings

The proposed scheme achieves significant model size reduction.

It maintains high inference accuracy after compression.

Compared to existing methods, it shows superior sparsity and performance trade-offs.

Abstract

Despite the growing availability of high-capacity computational platforms, implementation complexity still has been a great concern for the real-world deployment of neural networks. This concern is not exclusively due to the huge costs of state-of-the-art network architectures, but also due to the recent push towards edge intelligence and the use of neural networks in embedded applications. In this context, network compression techniques have been gaining interest due to their ability for reducing deployment costs while keeping inference accuracy at satisfactory levels. The present paper is dedicated to the development of a novel compression scheme for neural networks. To this end, a new form of $\ell_0$-norm-based regularization is firstly developed, which is capable of inducing strong sparseness in the network during training. Then, targeting the smaller weights of the trained network with pruning techniques, smaller yet highly effective networks can be obtained. The proposed compression scheme also involves the use of $\ell_2$-norm regularization to avoid overfitting as well as fine tuning to improve the performance of the pruned network. Experimental results are presented aiming to show the effectiveness of the proposed scheme as well as to make comparisons with competing approaches.