Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data

TL;DR

This paper demonstrates that Uniform-based Discrete Diffusion Models function as associative memories with emergent creative abilities, and introduces a method to detect their memorization and generalization regimes via conditional entropy analysis.

Contribution

It reveals the associative memory behavior of UDDMs and proposes a practical entropy-based metric to distinguish memorization from generalization in generative models.

Findings

UDDMs behave as associative memories with basins of attraction.

A sharp transition from memorization to generalization is observed as training data size increases.

Conditional entropy effectively detects the memorization-generalization transition.

Abstract

When do language diffusion models memorize their training data, and how to quantitatively assess their true generative regime? We address these questions by showing that Uniform-based Discrete Diffusion Models (UDDMs) fundamentally behave as Associative Memories (AMs) $\textit{with emergent creative capabilities}$. The core idea of an AM is to reliably recover stored data points as $\textit{memories}$ by establishing distinct basins of attraction around them. Historically, models like Hopfield networks use an explicit energy function to guarantee these stable attractors. We broaden this perspective by leveraging the observation that energy is not strictly necessary, as basins of attraction can also be formed via conditional likelihood maximization. By evaluating token recovery of $\textit{training}$ and $\textit{test}$ examples, we identify in UDDMs a sharp memorization-to-generalization transition governed by the size of the training dataset: as it increases, basins around training examples shrink and basins around unseen test examples expand, until both later converge to the same level. Crucially, we can detect this transition using only the conditional entropy of predicted token sequences: memorization is characterized by vanishing conditional entropy, while in the generalization regime the conditional entropy of most tokens remains finite. Thus, conditional entropy offers a practical probe for the memorization-to-generalization transition in deployed models.