Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decomposition

TL;DR

This paper introduces a Hierarchical Tucker tensor decomposition method to compress RNNs, significantly improving model size reduction and accuracy compared to existing tensor-based approaches.

Contribution

The paper proposes a novel HT decomposition-based RNN compression method that enhances representation capability and achieves superior compression and accuracy.

Findings

HT-LSTM outperforms TT-LSTM, TR-LSTM, and BT-LSTM in compression ratio.

HT-LSTM maintains or improves test accuracy across datasets.

Hierarchical structure enhances model expressiveness.

Abstract

Recurrent Neural Networks (RNNs) have been widely used in sequence analysis and modeling. However, when processing high-dimensional data, RNNs typically require very large model sizes, thereby bringing a series of deployment challenges. Although the state-of-the-art tensor decomposition approaches can provide good model compression performance, these existing methods are still suffering some inherent limitations, such as restricted representation capability and insufficient model complexity reduction. To overcome these limitations, in this paper we propose to develop compact RNN models using Hierarchical Tucker (HT) decomposition. HT decomposition brings strong hierarchical structure to the decomposed RNN models, which is very useful and important for enhancing the representation capability. Meanwhile, HT decomposition provides higher storage and computational cost reduction than the existing tensor decomposition approaches for RNN compression. Our experimental results show that, compared with the state-of-the-art compressed RNN models, such as TT-LSTM, TR-LSTM and BT-LSTM, our proposed HT-based LSTM (HT-LSTM), consistently achieves simultaneous and significant increases in both compression ratio and test accuracy on different datasets.