Low-Perplexity LLM-Generated Sequences and Where To Find Them

TL;DR

This paper presents a systematic method to analyze low-perplexity sequences generated by LLMs, revealing insights into how training data influences model outputs and identifying the extent of data memorization.

Contribution

Introduces a pipeline for extracting and tracing low-perplexity sequences in LLM outputs to their training data sources, enhancing transparency and understanding of model behavior.

Findings

Many low-perplexity sequences cannot be traced to training data.

Some sequences are verbatim recalls from training sources.

The approach helps quantify the influence of training data on generated text.

Abstract

As Large Language Models (LLMs) become increasingly widespread, understanding how specific training data shapes their outputs is crucial for transparency, accountability, privacy, and fairness. To explore how LLMs leverage and replicate their training data, we introduce a systematic approach centered on analyzing low-perplexity sequences - high-probability text spans generated by the model. Our pipeline reliably extracts such long sequences across diverse topics while avoiding degeneration, then traces them back to their sources in the training data. Surprisingly, we find that a substantial portion of these low-perplexity spans cannot be mapped to the corpus. For those that do match, we quantify the distribution of occurrences across source documents, highlighting the scope and nature of verbatim recall and paving a way toward better understanding of how LLMs training data impacts their behavior.