Uncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Frontier AI Models

TL;DR

This paper investigates how large AI models memorize and leak sensitive data, revealing the phenomenon of latent memorization that can occur without repeated exposure, and proposes a diagnostic method to detect such hidden memorization.

Contribution

It introduces the concept of latent memorization in AI models, demonstrating its occurrence during training and developing a diagnostic test to detect hidden data leakage.

Findings

Memorization probability scales logarithmically with data frequency.

Latent memorization can occur without repeated data exposure.

A diagnostic test effectively uncovers hidden memorized sequences.

Abstract

Frontier AI systems are making transformative impacts across society, but such benefits are not without costs: models trained on web-scale datasets containing personal and private data raise profound concerns about data privacy and security. Language models are trained on extensive corpora including potentially sensitive or proprietary information, and the risk of data leakage - where the model response reveals pieces of such information - remains inadequately understood. Prior work has investigated what factors drive memorization and have identified that sequence complexity and the number of repetitions drive memorization. Here, we focus on the evolution of memorization over training. We begin by reproducing findings that the probability of memorizing a sequence scales logarithmically with the number of times it is present in the data. We next show that sequences which are apparently not memorized after the first encounter can be "uncovered" throughout the course of training even without subsequent encounters, a phenomenon we term "latent memorization". The presence of latent memorization presents a challenge for data privacy as memorized sequences may be hidden at the final checkpoint of the model but remain easily recoverable. To this end, we develop a diagnostic test relying on the cross entropy loss to uncover latent memorized sequences with high accuracy.