Fast Weight Long Short-Term Memory

TL;DR

This paper demonstrates that integrating fast weight associative memory with LSTM networks significantly enhances training speed and accuracy in complex memory tasks, revealing a beneficial synergy for recurrent neural networks.

Contribution

It introduces the novel combination of fast weight memory with LSTM networks, showing improved performance and training efficiency in associative retrieval tasks.

Findings

Faster training of LSTM with fast weights.

Lower test error in high-difficulty tasks.

Significant performance boost at complex memory tasks.

Abstract

Associative memory using fast weights is a short-term memory mechanism that substantially improves the memory capacity and time scale of recurrent neural networks (RNNs). As recent studies introduced fast weights only to regular RNNs, it is unknown whether fast weight memory is beneficial to gated RNNs. In this work, we report a significant synergy between long short-term memory (LSTM) networks and fast weight associative memories. We show that this combination, in learning associative retrieval tasks, results in much faster training and lower test error, a performance boost most prominent at high memory task difficulties.