Tensor train decompositions on recurrent networks

TL;DR

This paper advocates for using matrix product state (MPS) tensor trains to compress recurrent neural networks, especially LSTMs, offering better storage and inference efficiency, supported by theoretical and experimental evidence.

Contribution

It introduces MPS tensor trains as a superior method for RNN compression, highlighting their advantages over MPOs through analysis and NLP experiments.

Findings

MPS tensor trains outperform MPOs in storage efficiency.

MPS-based LSTM compression maintains performance with fewer parameters.

Theoretical analysis supports MPS advantages in RNN compression.

Abstract

Recurrent neural networks (RNN) such as long-short-term memory (LSTM) networks are essential in a multitude of daily live tasks such as speech, language, video, and multimodal learning. The shift from cloud to edge computation intensifies the need to contain the growth of RNN parameters. Current research on RNN shows that despite the performance obtained on convolutional neural networks (CNN), keeping a good performance in compressed RNNs is still a challenge. Most of the literature on compression focuses on CNNs using matrix product (MPO) operator tensor trains. However, matrix product state (MPS) tensor trains have more attractive features than MPOs, in terms of storage reduction and computing time at inference. We show that MPS tensor trains should be at the forefront of LSTM network compression through a theoretical analysis and practical experiments on NLP task.