$\ell_0$ Regularized Structured Sparsity Convolutional Neural Networks

TL;DR

This paper introduces a novel $\,\ell_0$ regularized structured sparsity method for CNNs that effectively reduces model size while maintaining high accuracy, using a new sparse group lasso variant and an iterative optimization approach.

Contribution

It proposes a new $\,\ell_0$ regularized structured sparsity technique for CNNs, combining $\,\ell_0$ and $\,\ell_{2,1}$ norms, with an algorithm for efficient training and sparsification.

Findings

Achieves high test accuracy with significant sparsity on LeNet-5 and Wide-ResNet.

Outperforms existing sparsification methods in accuracy and compression.

Effective on MNIST and CIFAR datasets.

Abstract

Deepening and widening convolutional neural networks (CNNs) significantly increases the number of trainable weight parameters by adding more convolutional layers and feature maps per layer, respectively. By imposing inter- and intra-group sparsity onto the weights of the layers during the training process, a compressed network can be obtained with accuracy comparable to a dense one. In this paper, we propose a new variant of sparse group lasso that blends the $\ell_0$ norm onto the individual weight parameters and the $\ell_{2,1}$ norm onto the output channels of a layer. To address the non-differentiability of the $\ell_0$ norm, we apply variable splitting resulting in an algorithm that consists of executing stochastic gradient descent followed by hard thresholding for each iteration. Numerical experiments are demonstrated on LeNet-5 and wide-residual-networks for MNIST and CIFAR 10/100, respectively. They showcase the effectiveness of our proposed method in attaining superior test accuracy with network sparsification on par with the current state of the art.