Tensor Decomposition for Compressing Recurrent Neural Network

TL;DR

This paper explores tensor decomposition techniques like CP, Tucker, and Tensor Train to compress Gated Recurrent Units in RNNs, aiming to reduce parameters while maintaining performance, with Tensor Train showing the best results.

Contribution

It introduces tensor decomposition methods for RNN compression, demonstrating their effectiveness and comparing different approaches on sequence modeling tasks.

Findings

Tensor Train-GRU outperforms other tensor decomposition methods.

Tensor decompositions significantly reduce RNN parameters.

Performance is maintained across various parameter sizes.

Abstract

In the machine learning fields, Recurrent Neural Network (RNN) has become a popular architecture for sequential data modeling. However, behind the impressive performance, RNNs require a large number of parameters for both training and inference. In this paper, we are trying to reduce the number of parameters and maintain the expressive power from RNN simultaneously. We utilize several tensor decompositions method including CANDECOMP/PARAFAC (CP), Tucker decomposition and Tensor Train (TT) to re-parameterize the Gated Recurrent Unit (GRU) RNN. We evaluate all tensor-based RNNs performance on sequence modeling tasks with a various number of parameters. Based on our experiment results, TT-GRU achieved the best results in a various number of parameters compared to other decomposition methods.