Associative Long Short-Term Memory

TL;DR

This paper introduces an associative memory extension to LSTM networks using complex-valued vectors, enabling faster learning and improved retrieval accuracy without increasing model size.

Contribution

It proposes a novel associative memory mechanism based on complex vectors that enhances LSTM performance by reducing retrieval noise and increasing redundancy.

Findings

Faster learning on memorization tasks

Reduced retrieval noise due to redundancy

Comparable or improved performance without extra parameters

Abstract

We investigate a new method to augment recurrent neural networks with extra memory without increasing the number of network parameters. The system has an associative memory based on complex-valued vectors and is closely related to Holographic Reduced Representations and Long Short-Term Memory networks. Holographic Reduced Representations have limited capacity: as they store more information, each retrieval becomes noisier due to interference. Our system in contrast creates redundant copies of stored information, which enables retrieval with reduced noise. Experiments demonstrate faster learning on multiple memorization tasks.