Memorization in Language Models through the Lens of Intrinsic Dimension

TL;DR

This paper explores how the intrinsic dimension of sequences in language models influences memorization, revealing that higher intrinsic dimension reduces the likelihood of memorization, especially in large, sparse models.

Contribution

It introduces the concept of intrinsic dimension as a geometric measure to understand and predict memorization in language models, highlighting its suppressive effect.

Findings

High intrinsic dimension sequences are less likely to be memorized.

Memorization is influenced by scale, exposure, and complexity interactions.

Intrinsic dimension acts as a suppressive signal for memorization.

Abstract

Language Models (LMs) are prone to memorizing parts of their data during training and unintentionally emitting them at generation time, raising concerns about privacy leakage and disclosure of intellectual property. While previous research has identified properties such as context length, parameter size, and duplication frequency, as key drivers of unintended memorization, little is known about how the latent structure modulates this rate of memorization. We investigate the role of Intrinsic Dimension (ID), a geometric proxy for the structural complexity of a sequence in latent space, in modulating memorization. Our findings suggest that ID acts as a suppressive signal for memorization: compared to low-ID sequences, high-ID sequences are less likely to be memorized, particularly in overparameterized models and under sparse exposure. These findings highlight the interaction between scale, exposure, and complexity in shaping memorization.