Understanding Unintended Memorization in Federated Learning

TL;DR

This paper investigates how federated learning components influence unintended memorization in models, finding that data clustering, federated averaging, and differential privacy significantly reduce memorization of sensitive data.

Contribution

It provides a formal analysis of how federated learning components impact unintended memorization, highlighting the roles of data clustering, federated averaging, and differential privacy.

Findings

Data clustering reduces memorization.

Federated averaging further decreases memorization.

Differential privacy leads to models with minimal memorization.

Abstract

Recent works have shown that generative sequence models (e.g., language models) have a tendency to memorize rare or unique sequences in the training data. Since useful models are often trained on sensitive data, to ensure the privacy of the training data it is critical to identify and mitigate such unintended memorization. Federated Learning (FL) has emerged as a novel framework for large-scale distributed learning tasks. However, it differs in many aspects from the well-studied central learning setting where all the data is stored at the central server. In this paper, we initiate a formal study to understand the effect of different components of canonical FL on unintended memorization in trained models, comparing with the central learning setting. Our results show that several differing components of FL play an important role in reducing unintended memorization. Specifically, we observe that the clustering of data according to users---which happens by design in FL---has a significant effect in reducing such memorization, and using the method of Federated Averaging for training causes a further reduction. We also show that training with a strong user-level differential privacy guarantee results in models that exhibit the least amount of unintended memorization.