Factorization tricks for LSTM networks

TL;DR

This paper introduces two parameter reduction techniques for LSTM networks—matrix factorization and matrix partitioning—that enable faster training and reduced model size while maintaining high performance.

Contribution

The paper proposes novel methods for reducing parameters in LSTM networks, improving training speed and efficiency without sacrificing accuracy.

Findings

Faster training of large LSTMs with fewer parameters.

Achieved near state-of-the-art perplexity with reduced model complexity.

Demonstrated effectiveness of the proposed methods on benchmark tasks.

Abstract

We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups. Both approaches allow us to train large LSTM networks significantly faster to the near state-of the art perplexity while using significantly less RNN parameters.