RotLSTM: Rotating Memories in Recurrent Neural Networks

TL;DR

RotLSTM introduces trainable rotation matrices to modify LSTM cell states, significantly improving performance on certain tasks by enhancing long-term dependency modeling.

Contribution

The paper proposes a novel modification to LSTM units using rotation matrices, which is a new approach to enhance memory capabilities in recurrent neural networks.

Findings

Improved performance on bAbI dataset tasks.

Rotation matrices enhance long-term dependency learning.

Demonstrates the effectiveness of memory rotation in RNNs.

Abstract

Long Short-Term Memory (LSTM) units have the ability to memorise and use long-term dependencies between inputs to generate predictions on time series data. We introduce the concept of modifying the cell state (memory) of LSTMs using rotation matrices parametrised by a new set of trainable weights. This addition shows significant increases of performance on some of the tasks from the bAbI dataset.