Differentiable Mask for Pruning Convolutional and Recurrent Networks

TL;DR

This paper introduces a differentiable mask technique for pruning both convolutional and recurrent neural networks, enabling effective model compression across vision and text-based architectures for multi-modal applications.

Contribution

The paper presents a novel differentiable mask method that generalizes pruning to various network components and architectures, including convolutional and recurrent models.

Findings

Effective pruning of weights, filters, and subnetworks.

Applicable to both vision and text-based models.

Enhances model compression for multi-modal learning.

Abstract

Pruning is one of the most effective model reduction techniques. Deep networks require massive computation and such models need to be compressed to bring them on edge devices. Most existing pruning techniques are focused on vision-based models like convolutional networks, while text-based models are still evolving. The emergence of multi-modal multi-task learning calls for a general method that works on vision and text architectures simultaneously. We introduce a \emph{differentiable mask}, that induces sparsity on various granularity to fill this gap. We apply our method successfully to prune weights, filters, subnetwork of a convolutional architecture, as well as nodes of a recurrent network.