Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings

TL;DR

This paper demonstrates that large language models can generate reversible sentence embeddings using a special memory token, enabling exact reconstruction of original text without changing model weights, across multiple languages and model sizes.

Contribution

Introduction of a memory token technique that allows LLMs to produce reversible sentence embeddings for the first time, without modifying model weights.

Findings

Llama 3.1 8B successfully reconstructs all tested sequences.

Reversible embeddings work across English and Spanish datasets.

Effective for sequences up to approximately 240 tokens.

Abstract

In this work, we observe an interesting phenomenon: it is possible to generate reversible sentence embeddings that allow an LLM to reconstruct the original text exactly, without modifying the model's weights. This is achieved by introducing a special memory token, whose embedding is optimized through training on a fixed sequence. When prompted with this embedding, the model reconstructs the fixed sequence exactly. We evaluate this phenomenon across English and Spanish datasets, sequences of up to approximately 240 tokens, and model scales ranging from 100M to 8B parameters. Notably, Llama 3.1 8B successfully reconstructs all tested sequences. Our findings highlight an interesting capability of LLMs and suggest potential applications in memory-based retrieval, compression, and controlled text generation.