PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMs

TL;DR

This paper introduces PII-Scope, a benchmark for evaluating PII extraction attacks on LLMs, revealing that sophisticated attack strategies significantly increase leakage and finetuned models are more vulnerable.

Contribution

We develop a comprehensive benchmark and study to evaluate PII extraction attacks, highlighting the impact of advanced adversarial strategies and model finetuning on leakage.

Findings

Sophisticated attacks can increase PII extraction rates by up to five times.

Finetuned models are more vulnerable to PII leakage than pretrained models.

Existing single-query attacks underestimate true PII leakage.

Abstract

In this work, we introduce PII-Scope, a comprehensive benchmark designed to evaluate state-of-the-art methodologies for PII extraction attacks targeting LLMs across diverse threat settings. Our study provides a deeper understanding of these attacks by uncovering several hyperparameters (e.g., demonstration selection) crucial to their effectiveness. Building on this understanding, we extend our study to more realistic attack scenarios, exploring PII attacks that employ advanced adversarial strategies, including repeated and diverse querying, and leveraging iterative learning for continual PII extraction. Through extensive experimentation, our results reveal a notable underestimation of PII leakage in existing single-query attacks. In fact, we show that with sophisticated adversarial capabilities and a limited query budget, PII extraction rates can increase by up to fivefold when targeting the pretrained model. Moreover, we evaluate PII leakage on finetuned models, showing that they are more vulnerable to leakage than pretrained models. Overall, our work establishes a rigorous empirical benchmark for PII extraction attacks in realistic threat scenarios and provides a strong foundation for developing effective mitigation strategies.