Layer Sparsity in Neural Networks

TL;DR

This paper introduces the concept of layer sparsity in neural networks, proposing regularization techniques to create more efficient and accurate models aligned with current deep learning practices.

Contribution

It formulates a new layer sparsity notion and develops regularization and refitting schemes to improve neural network efficiency and accuracy.

Findings

Layer sparsity effectively reduces network complexity.

Regularization schemes improve model compactness.

Enhanced accuracy with sparser networks.

Abstract

Sparsity has become popular in machine learning, because it can save computational resources, facilitate interpretations, and prevent overfitting. In this paper, we discuss sparsity in the framework of neural networks. In particular, we formulate a new notion of sparsity that concerns the networks' layers and, therefore, aligns particularly well with the current trend toward deep networks. We call this notion layer sparsity. We then introduce corresponding regularization and refitting schemes that can complement standard deep-learning pipelines to generate more compact and accurate networks.