A Multi-Perspective Analysis of Memorization in Large Language Models

TL;DR

This paper investigates the causes, dynamics, and characteristics of memorization in large language models, revealing how model size and context influence memorization and demonstrating the ability to predict memorization from context.

Contribution

It provides a comprehensive analysis of memorization in LLMs, including experiments, embedding analysis, and a predictive model, extending understanding beyond previous studies.

Findings

Memorization relates to model size, continuation size, and context size.

Embedding space analysis shows distribution differences for memorized vs. unmemorized content.

A trained model can predict memorization based on context.

Abstract

Large Language Models (LLMs), trained on massive corpora with billions of parameters, show unprecedented performance in various fields. Though surprised by their excellent performances, researchers also noticed some special behaviors of those LLMs. One of those behaviors is memorization, in which LLMs can generate the same content used to train them. Though previous research has discussed memorization, the memorization of LLMs still lacks explanation, especially the cause of memorization and the dynamics of generating them. In this research, we comprehensively discussed memorization from various perspectives and extended the discussion scope to not only just the memorized content but also less and unmemorized content. Through various studies, we found that: (1) Through experiments, we revealed the relation of memorization between model size, continuation size, and context size. Further, we showed how unmemorized sentences transition to memorized sentences. (2) Through embedding analysis, we showed the distribution and decoding dynamics across model size in embedding space for sentences with different memorization scores. The n-gram statistics analysis presents d (3) An analysis over n-gram and entropy decoding dynamics discovered a boundary effect when the model starts to generate memorized sentences or unmemorized sentences. (4)We trained a Transformer model to predict the memorization of different models, showing that it is possible to predict memorizations by context.