Make $\ell_1$ Regularization Effective in Training Sparse CNN

TL;DR

This paper demonstrates how to effectively apply $ abla$ regularization to train sparse CNNs by replacing SGD with a regularized dual averaging method, achieving high sparsity without accuracy loss.

Contribution

It introduces a modified training algorithm based on RDA for $ abla$ regularization in CNNs, overcoming previous incompatibility issues.

Findings

Achieves 95% sparsity in ResNet18 on CIFAR-10.

Outperforms existing weight pruning methods in sparsity.

Maintains high accuracy with sparse models.

Abstract

Compressed Sensing using $\ell_1$ regularization is among the most powerful and popular sparsification technique in many applications, but why has it not been used to obtain sparse deep learning model such as convolutional neural network (CNN)? This paper is aimed to provide an answer to this question and to show how to make it work. We first demonstrate that the commonly used stochastic gradient decent (SGD) and variants training algorithm is not an appropriate match with $\ell_1$ regularization and then replace it with a different training algorithm based on a regularized dual averaging (RDA) method. RDA was originally designed specifically for convex problem, but with new theoretical insight and algorithmic modifications (using proper initialization and adaptivity), we have made it an effective match with $\ell_1$ regularization to achieve a state-of-the-art sparsity for CNN compared to other weight pruning methods without compromising accuracy (achieving 95\% sparsity for ResNet18 on CIFAR-10, for example).