MeMo: Towards Language Models with Associative Memory Mechanisms

TL;DR

This paper introduces MeMo, a new language model architecture that explicitly memorizes text sequences using layered associative memories, enhancing transparency and enabling model editing like forgetting specific texts.

Contribution

Proposes MeMo, a novel architecture for language models that directly memorizes text sequences with layered associative memories, shifting from learning-based memorization.

Findings

MeMo demonstrates strong memorization capabilities in various configurations.

The architecture offers transparency and facilitates model editing such as forgetting.

Experimental results validate the effectiveness of MeMo in memorization tasks.

Abstract

Memorization is a fundamental ability of Transformer-based Large Language Models, achieved through learning. In this paper, we propose a paradigm shift by designing an architecture to memorize text directly, bearing in mind the principle that memorization precedes learning. We introduce MeMo, a novel architecture for language modeling that explicitly memorizes sequences of tokens in layered associative memories. By design, MeMo offers transparency and the possibility of model editing, including forgetting texts. We experimented with the MeMo architecture, showing the memorization power of the one-layer and the multi-layer configurations.