Towards Extremely Compact RNNs for Video Recognition with Fully Decomposed Hierarchical Tucker Structure

TL;DR

This paper introduces a fully decomposed hierarchical Tucker (FDHT) structure for RNNs, enabling extremely compact models that maintain high accuracy, particularly for video recognition tasks, with significantly fewer parameters than existing methods.

Contribution

The paper proposes a novel fully decomposition approach using hierarchical Tucker structure for RNNs, achieving superior compression and accuracy over prior tensor decomposition methods.

Findings

FDHT-LSTM achieves high accuracy with only a few thousand parameters.

Our method reduces model size by over 10,000 times compared to previous models.

FDHT-LSTM outperforms state-of-the-art compressed RNNs in both size and accuracy.

Abstract

Recurrent Neural Networks (RNNs) have been widely used in sequence analysis and modeling. However, when processing high-dimensional data, RNNs typically require very large model sizes, thereby bringing a series of deployment challenges. Although various prior works have been proposed to reduce the RNN model sizes, executing RNN models in resource-restricted environments is still a very challenging problem. In this paper, we propose to develop extremely compact RNN models with fully decomposed hierarchical Tucker (FDHT) structure. The HT decomposition does not only provide much higher storage cost reduction than the other tensor decomposition approaches but also brings better accuracy performance improvement for the compact RNN models. Meanwhile, unlike the existing tensor decomposition-based methods that can only decompose the input-to-hidden layer of RNNs, our proposed fully decomposition approach enables the comprehensive compression for the entire RNN models with maintaining very high accuracy. Our experimental results on several popular video recognition datasets show that our proposed fully decomposed hierarchical tucker-based LSTM (FDHT-LSTM) is extremely compact and highly efficient. To the best of our knowledge, FDHT-LSTM, for the first time, consistently achieves very high accuracy with only few thousand parameters (3,132 to 8,808) on different datasets. Compared with the state-of-the-art compressed RNN models, such as TT-LSTM, TR-LSTM and BT-LSTM, our FDHT-LSTM simultaneously enjoys both order-of-magnitude (3,985x to 10,711x) fewer parameters and significant accuracy improvement (0.6% to 12.7%).