Memorization Sinks: Isolating Memorization during LLM Training

TL;DR

This paper introduces MemSinks, a novel method that isolates memorized sequences in large language models by design, enabling easier removal of memorized content without harming the model's general language abilities.

Contribution

The paper proposes MemSinks, a new paradigm that uses sequence identifiers to isolate memorization in LLMs, demonstrated at billion-parameter scale with effective memorization removal and preserved generalization.

Findings

MemSinks effectively isolates memorized sequences.

MemSinks allows removal of memorization without harming language capabilities.

First proof-of-concept demonstrating simultaneous memorization isolation and generalization.

Abstract

Large language models are susceptible to memorizing repeated sequences, posing privacy and copyright concerns. A popular mitigation strategy is to remove memorized information from specific neurons post-hoc. However, such approaches have shown limited success so far. In a controlled setting, we show that the memorization of natural sequences (those that resemble linguistically plausible text) become mechanistically entangled with general language abilities, thereby becoming challenging to remove post-hoc. In this work, we put forward a new paradigm of MemSinks that promotes isolation of memorization by design. We leverage a sequence identifier that activates a unique set of memorization neurons for each sequence across repetitions. By analyzing the dynamics of learning and forgetting, we argue that MemSinks facilitates isolation of memorized content, making it easier to remove without compromising general language capabilities. We implement MemSinks at the billion-parameter and billion-token scale, and observe both effective isolation and strong generalization. To our knowledge, this is the first proof-of-concept on real data demonstrating that simultaneous generalization and isolation is achievable. We open-source our code at http://github.com/grghosal/MemSinks.