Memorization in Fine-Tuned Large Language Models

TL;DR

This paper explores how fine-tuning large language models, especially in sensitive domains like medicine, affects their tendency to memorize training data, highlighting the impact of different fine-tuning techniques on privacy risks.

Contribution

It provides a detailed analysis of factors influencing memorization in fine-tuned LLMs, including the roles of specific weight matrices and fine-tuning parameters, with practical insights for privacy-aware model adaptation.

Findings

Value and Output matrices contribute more to memorization.

Lower perplexity correlates with increased memorization.

Higher LoRA ranks increase memorization with diminishing returns.

Abstract

This study investigates the mechanisms and factors influencing memorization in fine-tuned large language models (LLMs), with a focus on the medical domain due to its privacy-sensitive nature. We examine how different aspects of the fine-tuning process affect a model's propensity to memorize training data, using the PHEE dataset of pharmacovigilance events. Our research employs two main approaches: a membership inference attack to detect memorized data, and a generation task with prompted prefixes to assess verbatim reproduction. We analyze the impact of adapting different weight matrices in the transformer architecture, the relationship between perplexity and memorization, and the effect of increasing the rank in low-rank adaptation (LoRA) fine-tuning. Key findings include: (1) Value and Output matrices contribute more significantly to memorization compared to Query and Key matrices; (2) Lower perplexity in the fine-tuned model correlates with increased memorization; (3) Higher LoRA ranks lead to increased memorization, but with diminishing returns at higher ranks. These results provide insights into the trade-offs between model performance and privacy risks in fine-tuned LLMs. Our findings have implications for developing more effective and responsible strategies for adapting large language models while managing data privacy concerns.