Do RNN and LSTM have Long Memory?

TL;DR

This paper investigates whether RNNs and LSTMs truly possess long-term memory, proving they generally do not from a statistical perspective, and introduces a new definition for long memory networks.

Contribution

It provides a theoretical proof that RNNs and LSTMs lack long memory and proposes a new polynomial decay-based definition for long memory networks.

Findings

RNNs and LSTMs do not have long memory under statistical analysis.

Minimal modifications can convert RNNs and LSTMs into long memory networks.

Modified models outperform standard ones in modeling long-term dependencies.

Abstract

The LSTM network was proposed to overcome the difficulty in learning long-term dependence, and has made significant advancements in applications. With its success and drawbacks in mind, this paper raises the question - do RNN and LSTM have long memory? We answer it partially by proving that RNN and LSTM do not have long memory from a statistical perspective. A new definition for long memory networks is further introduced, and it requires the model weights to decay at a polynomial rate. To verify our theory, we convert RNN and LSTM into long memory networks by making a minimal modification, and their superiority is illustrated in modeling long-term dependence of various datasets.