Bayesian Compression for Natural Language Processing

TL;DR

This paper introduces a Bayesian sparsification method for RNNs in NLP that significantly reduces model size by compressing parameters and filtering unnecessary words, with interpretable word selection.

Contribution

It presents a novel Bayesian sparsification technique for RNNs that enables large-scale compression and interpretable vocabulary filtering without extensive hyperparameter tuning.

Findings

RNN models can be compressed dozens to hundreds of times.

Vocabulary filtering leads to more compact models with interpretable word choices.

The method does not require time-consuming hyperparameter tuning.

Abstract

In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian sparsification technique for RNNs which allows compressing the RNN dozens or hundreds of times without time-consuming hyperparameters tuning. We also generalize the model for vocabulary sparsification to filter out unnecessary words and compress the RNN even further. We show that the choice of the kept words is interpretable. Code is available on github: https://github.com/tipt0p/SparseBayesianRNN