Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models

TL;DR

This paper demonstrates that federated large language models can unintentionally leak private data through simple extraction methods, highlighting significant privacy risks and providing a benchmark for future privacy-preserving techniques.

Contribution

It introduces effective data extraction strategies to reveal privacy leaks in FedLLMs and establishes a benchmark dataset and evaluation framework for privacy risk assessment.

Findings

Up to 56.6% of victim-exclusive PII can be recovered.

Names, addresses, and birthdays are highly vulnerable.

Proposes simple extraction methods that exploit memorization in FedLLMs.

Abstract

Federated large language models (FedLLMs) enable cross-silo collaborative training among institutions while preserving data locality, making them appealing for privacy-sensitive domains such as law, finance, and healthcare. However, the memorization behavior of LLMs can lead to privacy risks that may cause cross-client data leakage. In this work, we study the threat of cross-client data extraction, where a semi-honest participant attempts to recover personally identifiable information (PII) memorized from other clients' data. We propose three simple yet effective extraction strategies that leverage contextual prefixes from the attacker's local data, including frequency-based prefix sampling and local fine-tuning to amplify memorization. To evaluate these attacks, we construct a Chinese legal-domain dataset with fine-grained PII annotations consistent with CPIS, GDPR, and CCPA standards, and assess extraction performance using two metrics: coverage and efficiency. Experimental results show that our methods can recover up to 56.6% of victim-exclusive PII, where names, addresses, and birthdays are particularly vulnerable. These findings highlight concrete privacy risks in FedLLMs and establish a benchmark and evaluation framework for future research on privacy-preserving federated learning. Code and data are available at https://github.com/SMILELab-FL/FedPII.