L0 Regularization Based Neural Network Design and Compression

TL;DR

This paper explores using L0 regularization to reduce the complexity of deep neural networks, aiming to improve interpretability, robustness, and efficiency, with practical applications demonstrated on MNIST and signal classification tasks.

Contribution

It introduces a method of applying L0 regularization to DNNs to effectively control complexity and interpret trade-offs, highlighting the importance of the tradeoff knee point.

Findings

L0 regularization captures input saliency.

Tradeoff curve exhibits a knee point indicating optimal complexity.

Regularization improves interpretability and robustness.

Abstract

We consider complexity of Deep Neural Networks (DNNs) and their associated massive over-parameterization. Such over-parametrization may entail susceptibility to adversarial attacks, loss of interpretability and adverse Size, Weight and Power - Cost (SWaP-C) considerations. We ask if there are methodical ways (regularization) to reduce complexity and how can we interpret trade-off between desired metric and complexity of DNN. Reducing complexity is directly applicable to scaling of AI applications to real world problems (especially for off-the-cloud applications). We show that presence and evaluation of the knee of the tradeoff curve. We apply a form of L0 regularization to MNIST data and signal modulation classifications. We show that such regularization captures saliency in the input space as well.