Teach LLMs to Phish: Stealing Private Information from Language Models

TL;DR

This paper introduces 'neural phishing,' a practical attack method that can extract sensitive personal information from large language models trained on private data, with high success rates, even with minimal data insertion.

Contribution

The authors propose a novel data extraction attack called neural phishing that effectively retrieves private information from language models with limited prior knowledge.

Findings

Attack success rates up to 50%

Effective with only tens of benign sentences inserted

Requires minimal prior knowledge about user data

Abstract

When large language models are trained on private data, it can be a significant privacy risk for them to memorize and regurgitate sensitive information. In this work, we propose a new practical data extraction attack that we call "neural phishing". This attack enables an adversary to target and extract sensitive or personally identifiable information (PII), e.g., credit card numbers, from a model trained on user data with upwards of 10% attack success rates, at times, as high as 50%. Our attack assumes only that an adversary can insert as few as 10s of benign-appearing sentences into the training dataset using only vague priors on the structure of the user data.