Recovering Private Text in Federated Learning of Language Models

TL;DR

This paper introduces FILM, a novel attack method demonstrating the feasibility of recovering private text data from federated learning of language models, highlighting privacy vulnerabilities and proposing effective defenses.

Contribution

The paper presents the first successful method for recovering text from large batch sizes in federated learning and evaluates defense strategies to mitigate this privacy risk.

Findings

High-fidelity text recovery from large batch sizes

Effective defense via fine-tuning pre-trained models without updating embeddings

Gradient pruning and DPSGD reduce attack success but impact utility

Abstract

Federated learning allows distributed users to collaboratively train a model while keeping each user's data private. Recently, a growing body of work has demonstrated that an eavesdropping attacker can effectively recover image data from gradients transmitted during federated learning. However, little progress has been made in recovering text data. In this paper, we present a novel attack method FILM for federated learning of language models (LMs). For the first time, we show the feasibility of recovering text from large batch sizes of up to 128 sentences. Unlike image-recovery methods that are optimized to match gradients, we take a distinct approach that first identifies a set of words from gradients and then directly reconstructs sentences based on beam search and a prior-based reordering strategy. We conduct the FILM attack on several large-scale datasets and show that it can successfully reconstruct single sentences with high fidelity for large batch sizes and even multiple sentences if applied iteratively. We evaluate three defense methods: gradient pruning, DPSGD, and a simple approach to freeze word embeddings that we propose. We show that both gradient pruning and DPSGD lead to a significant drop in utility. However, if we fine-tune a public pre-trained LM on private text without updating word embeddings, it can effectively defend the attack with minimal data utility loss. Together, we hope that our results can encourage the community to rethink the privacy concerns of LM training and its standard practices in the future.