Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs

TL;DR

This paper introduces Randomized Masked Fine-Tuning (RMFT), a new method to reduce PII memorization in LLMs with minimal performance loss, validated on the Enron Email Dataset with significant privacy improvements.

Contribution

The paper proposes RMFT, a novel privacy-preserving fine-tuning technique, and introduces MaxTER, a framework for evaluating privacy-utility tradeoffs in LLMs.

Findings

RMFT reduces PII memorization by over 80%.

RMFT maintains model performance with only 5.73% perplexity increase.

RMFT outperforms deduplication methods in privacy-utility tradeoff metrics.

Abstract

The current literature on memorization in Natural Language Models, especially Large Language Models (LLMs), poses severe security and privacy risks, as models tend to memorize personally identifying information (PIIs) from training data. We introduce Randomized Masked Fine-Tuning (RMFT), a novel privacy-preserving fine-tuning technique that reduces PII memorization while minimizing performance impact. Using the Enron Email Dataset, we demonstrate that RMFT achieves an 80.81% reduction in Total Extraction Rate and 80.17% reduction in Seen Extraction Rate compared to baseline fine-tuning, outperforming deduplication methods while maintaining only a 5.73% increase in perplexity. We present MaxTER, a Pareto-optimal evaluation framework for assessing privacy-utility tradeoffs, and show the performance of RMFT vs Deduplication by Area Under The Response Curve (AURC) metric.