Embedding Differentiable Sparsity into Deep Neural Network

TL;DR

This paper introduces a method to embed differentiable sparsity into deep neural networks, enabling simultaneous learning of network structure and weights with exact zero parameters during training.

Contribution

It presents a novel approach that allows neural networks to learn both sparse structures and weights simultaneously through differentiable sparsity embedding.

Findings

Supports both structured and unstructured sparsity

Allows exact zero parameters during training

Enables simultaneous learning of structure and weights

Abstract

In this paper, we propose embedding sparsity into the structure of deep neural networks, where model parameters can be exactly zero during training with the stochastic gradient descent. Thus, it can learn the sparsified structure and the weights of networks simultaneously. The proposed approach can learn structured as well as unstructured sparsity.