Improving Neural Language Models with a Continuous Cache

TL;DR

This paper introduces a continuous cache mechanism for neural language models, enhancing their ability to adapt predictions based on recent context, with improved performance over existing memory-augmented models.

Contribution

It presents a simplified, scalable memory extension for neural language models that effectively leverages recent history, bridging neural and count-based cache models.

Findings

Significant performance improvements over recent memory-augmented models

Efficient scalability to large memory sizes

Effective adaptation to recent context in language modeling

Abstract

We propose an extension to neural network language models to adapt their prediction to the recent history. Our model is a simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation. This mechanism is very efficient and scales to very large memory sizes. We also draw a link between the use of external memory in neural network and cache models used with count based language models. We demonstrate on several language model datasets that our approach performs significantly better than recent memory augmented networks.