Measuring Non-Adversarial Reproduction of Training Data in Large Language Models

TL;DR

This paper investigates the extent of non-adversarial memorization in large language models, revealing that a significant portion of their outputs can overlap with training data, and explores prompting strategies to mitigate this reproduction.

Contribution

It introduces the concept of non-adversarial reproduction, quantifies its prevalence in language models, and evaluates prompting strategies to reduce data overlap.

Findings

Up to 15% of model outputs overlap with training data.

In worst cases, 100% of generated content matches online data.

Prompting strategies can reduce but not eliminate non-adversarial reproduction.

Abstract

Large language models memorize parts of their training data. Memorizing short snippets and facts is required to answer questions about the world and to be fluent in any language. But models have also been shown to reproduce long verbatim sequences of memorized text when prompted by a motivated adversary. In this work, we investigate an intermediate regime of memorization that we call non-adversarial reproduction, where we quantify the overlap between model responses and pretraining data when responding to natural and benign prompts. For a variety of innocuous prompt categories (e.g., writing a letter or a tutorial), we show that up to 15% of the text output by popular conversational language models overlaps with snippets from the Internet. In worst cases, we find generations where 100% of the content can be found exactly online. For the same tasks, we find that human-written text has far less overlap with Internet data. We further study whether prompting strategies can close this reproduction gap between models and humans. While appropriate prompting can reduce non-adversarial reproduction on average, we find that mitigating worst-case reproduction of training data requires stronger defenses -- even for benign interactions.