Deep sequence models tend to memorize geometrically; it is unclear why

TL;DR

Deep sequence models develop a form of geometric memory that encodes global relationships, enabling complex reasoning tasks to be simplified into easy navigation, which challenges traditional associative memory views.

Contribution

The paper introduces the concept of geometric memory in deep sequence models, contrasting it with associative memory, and analyzes its origins and implications for neural embedding geometries.

Findings

Models encode global relationships as geometric memory.

Geometric memory simplifies complex reasoning into navigation tasks.

Spectral bias contributes to the emergence of geometric memory.

Abstract

Deep sequence models are said to store atomic facts predominantly in the form of associative memory: a brute-force lookup of co-occurring entities. We identify a dramatically different form of storage of atomic facts that we term as geometric memory. Here, the model has synthesized embeddings encoding novel global relationships between all entities, including ones that do not co-occur in training. Such storage is powerful: for instance, we show how it transforms a hard reasoning task involving an $\ell$-fold composition into an easy-to-learn $1$-step navigation task. From this phenomenon, we extract fundamental aspects of neural embedding geometries that are hard to explain. We argue that the rise of such a geometry, as against a lookup of local associations, cannot be straightforwardly attributed to typical supervisory, architectural, or optimizational pressures. Counterintuitively, a geometry is learned even when it is more complex than the brute-force lookup. Then, by analyzing a connection to Node2Vec, we demonstrate how the geometry stems from a spectral bias that -- in contrast to prevailing theories -- indeed arises naturally despite the lack of various pressures. This analysis also points out to practitioners a visible headroom to make Transformer memory more strongly geometric. We hope the geometric view of parametric memory encourages revisiting the default intuitions that guide researchers in areas like knowledge acquisition, capacity, discovery, and unlearning.