Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

TL;DR

This paper investigates how fine-tuned large language models unintentionally memorize and leak sensitive personal information, analyzing factors affecting this behavior and benchmarking privacy-preserving methods to mitigate risks.

Contribution

It systematically studies PII memorization in fine-tuned LLMs, evaluates factors influencing leakage, and benchmarks privacy-preserving techniques for better privacy-utility trade-offs.

Findings

Post-training methods offer better privacy-utility balance.

Differential privacy reduces leakage but may cause training instability.

Memorization of PII remains a persistent challenge in fine-tuned LLMs.

Abstract

Fine-tuning Large Language Models (LLMs) on sensitive datasets carries a substantial risk of unintended memorization and leakage of Personally Identifiable Information (PII), which can violate privacy regulations and compromise individual safety. In this work, we systematically investigate a critical and underexplored vulnerability: the exposure of PII that appears only in model inputs, not in training targets. Using both synthetic and real-world datasets, we design controlled extraction probes to quantify unintended PII memorization and study how factors such as language, PII frequency, task type, and model size influence memorization behavior. We further benchmark four privacy-preserving approaches including differential privacy, machine unlearning, regularization, and preference alignment, evaluating their trade-offs between privacy and task performance. Our results show that post-training methods generally provide more consistent privacy-utility trade-offs, while differential privacy achieves strong reduction in leakage in specific settings, although it can introduce training instability. These findings highlight the persistent challenge of memorization in fine-tuned LLMs and emphasize the need for robust, scalable privacy-preserving techniques.