Learning Intrinsic Sparse Structures within Long Short-Term Memory

TL;DR

This paper introduces Intrinsic Sparse Structures (ISS) within LSTM units to enable effective model compression, maintaining regularity and performance while significantly reducing model size and increasing speed.

Contribution

The paper proposes a novel ISS method that ensures consistent dimension reduction in LSTM components, enabling efficient compression without invalidating units.

Findings

Achieved 10.59x speedup on language modeling without accuracy loss.

Successfully compressed models to 2.69M weights for SQuAD QA task.

Extended the approach to non-LSTM RNNs like RHNs.

Abstract

Model compression is significant for the wide adoption of Recurrent Neural Networks (RNNs) in both user devices possessing limited resources and business clusters requiring quick responses to large-scale service requests. This work aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs. Independently reducing the sizes of basic structures can result in inconsistent dimensions among them, and consequently, end up with invalid LSTM units. To overcome the problem, we propose Intrinsic Sparse Structures (ISS) in LSTMs. Removing a component of ISS will simultaneously decrease the sizes of all basic structures by one and thereby always maintain the dimension consistency. By learning ISS within LSTM units, the obtained LSTMs remain regular while having much smaller basic structures. Based on group Lasso regularization, our method achieves 10.59x speedup without losing any perplexity of a language modeling of Penn TreeBank dataset. It is also successfully evaluated through a compact model with only 2.69M weights for machine Question Answering of SQuAD dataset. Our approach is successfully extended to non- LSTM RNNs, like Recurrent Highway Networks (RHNs). Our source code is publicly available at https://github.com/wenwei202/iss-rnns