Exploring Sparsity in Recurrent Neural Networks

TL;DR

This paper introduces a pruning technique for RNNs that reduces model size by up to 90% and accelerates inference, enabling efficient deployment on resource-constrained devices without significant loss of accuracy.

Contribution

The paper presents a method to prune RNN weights during training, achieving high sparsity and speed-up while maintaining accuracy, which is novel for recurrent neural network deployment.

Findings

Model size reduced by up to 90%

Inference speed improved by 2x to 7x

Accuracy remains close to original dense networks

Abstract

Recurrent Neural Networks (RNN) are widely used to solve a variety of problems and as the quantity of data and the amount of available compute have increased, so have model sizes. The number of parameters in recent state-of-the-art networks makes them hard to deploy, especially on mobile phones and embedded devices. The challenge is due to both the size of the model and the time it takes to evaluate it. In order to deploy these RNNs efficiently, we propose a technique to reduce the parameters of a network by pruning weights during the initial training of the network. At the end of training, the parameters of the network are sparse while accuracy is still close to the original dense neural network. The network size is reduced by 8x and the time required to train the model remains constant. Additionally, we can prune a larger dense network to achieve better than baseline performance while still reducing the total number of parameters significantly. Pruning RNNs reduces the size of the model and can also help achieve significant inference time speed-up using sparse matrix multiply. Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around 2x to 7x.