Hierarchical Memorization in Large Language Models: Evidence from Citation Generation

TL;DR

This study investigates how large language models memorize and reproduce academic citations, revealing hierarchical and threshold-based patterns of memorization influenced by citation frequency and data redundancy.

Contribution

It provides empirical evidence of hierarchical memorization in LLMs, showing how different citation metadata are recalled at varying redundancy levels and thresholds.

Findings

Factual accuracy scales with citation count and varies across domains.

Memorization exhibits thresholds at approximately 90 and 1,200 citations.

Memorization is hierarchical, with certain metadata recalled earlier than others.

Abstract

Large language models (LLMs) generate fluent text across a wide range of tasks, but the fabrication of non-existent academic citations remains a critical and well-documented failure mode. Building on prior work that frames hallucination and verbatim memorization as outcomes of the same probabilistic process, this study uses citation count as a proxy for training data redundancy and asks how this redundancy is internally structured within a single bibliographic record. Using GPT-4.1, we generated and manually verified 100 citations across twenty computer-science domains, measuring factual fidelity via cosine similarity against authentic metadata. We find that (i) factual accuracy varies substantially across domains and scales log-linearly with citation count, (ii) the model crosses two empirically identifiable thresholds; an inflection around 90 citations and a saturation point near 1,200 citations beyond which records are reproduced nearly verbatim, (iii) memorization is hierarchical, with titles and first authors recalled earliest while venues and numeric fields require far greater redundancy and publication years remain essentially unlearned, and (iv) even highly cited records can be conflated when their titles and authors overlap, an effect interpretable as spurious-attractor interference. Memorization in LLMs is therefore not a binary on/off state but a graduated, hierarchically layered phenomenon shaped by the uneven distribution of knowledge in the pretraining corpus.