Prior Aware Memorization: An Efficient Metric for Distinguishing Memorization from Generalization in Large Language Models

TL;DR

This paper introduces Prior-Aware Memorization, a lightweight, training-free metric to distinguish genuine memorization from common pattern generation in large language models, addressing privacy and security concerns.

Contribution

It proposes a theoretically grounded criterion that effectively differentiates memorized data from statistically common sequences without retraining models.

Findings

55% to 90% of sequences labeled as memorized are statistically common

Approximately 40% of sequences in the dataset show common-pattern behavior despite low frequency

The metric highlights the importance of considering model priors in leakage assessment

Abstract

Training data leakage from Large Language Models (LLMs) raises serious concerns related to privacy, security, and copyright compliance. A central challenge in assessing this risk is distinguishing genuine memorization of training data from the generation of statistically common sequences. Existing approaches to measuring memorization often conflate these phenomena, labeling outputs as memorized even when they arise from generalization over common patterns. Counterfactual Memorization provides a principled solution by comparing models trained with and without a target sequence, but its reliance on retraining multiple baseline models makes it computationally expensive and impractical at scale. This work introduces Prior-Aware Memorization, a theoretically grounded, lightweight and training-free criterion for identifying genuine memorization in LLMs. The key idea is to evaluate whether a candidate suffix is strongly associated with its specific training prefix or whether it appears with high probability across many unrelated prompts due to statistical commonality. We evaluate this metric on text from the training corpora of two pre-trained models, LLaMA and OPT, using both long sequences (to simulate copyright risks) and named entities (to simulate PII leakage). Our results show that between 55% and 90% of sequences previously labeled as memorized are in fact statistically common. Similar findings hold for the SATML training data extraction challenge dataset, where roughly 40% of sequences exhibit common-pattern behavior despite appearing only once in the training data. These results demonstrate that low frequency alone is insufficient evidence of memorization and highlight the importance of accounting for model priors when assessing leakage.