Neural Network Reduction with Guided Regularizers

TL;DR

This paper introduces Guided Regularization, a novel method that prioritizes certain neural network weights during training to facilitate more effective and targeted pruning of unnecessary units, improving model sparsity without performance loss.

Contribution

The paper proposes Guided Regularization, a new regularization technique that enhances neural network pruning by guiding the importance of specific units during training.

Findings

Effective pruning of neural networks demonstrated

Maintains performance after pruning

Outperforms traditional regularization methods

Abstract

Regularization techniques such as $\mathcal{L}_1$ and $\mathcal{L}_2$ regularizers are effective in sparsifying neural networks (NNs). However, to remove a certain neuron or channel in NNs, all weight elements related to that neuron or channel need to be prunable, which is not guaranteed by traditional regularization. This paper proposes a simple new approach named "Guided Regularization" that prioritizes the weights of certain NN units more than others during training, which renders some of the units less important and thus, prunable. This is different from the scattered sparsification of $\mathcal{L}_1$ and $\mathcal{L}_2$ regularizers where the the components of a weight matrix that are zeroed out can be located anywhere. The proposed approach offers a natural reduction of NN in the sense that a model is being trained while also neutralizing unnecessary units. We empirically demonstrate that our proposed method is effective in pruning NNs while maintaining performance.