Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data

TL;DR

This paper develops a theoretical framework to understand how differentially private training with DP-SGD affects memorization and generalization on long-tailed data, revealing increased test error on rare samples due to gradient clipping and noise.

Contribution

It introduces the first theoretical analysis of DP-SGD on long-tailed data, explaining the impact of privacy mechanisms on memorization and generalization.

Findings

DP-SGD increases test error on long-tailed subpopulations.

Gradient clipping and noise reduce memorization of rare samples.

Experimental validation supports theoretical insights.

Abstract

Recent research shows that modern deep learning models achieve high predictive accuracy partly by memorizing individual training samples. Such memorization raises serious privacy concerns, motivating the widespread adoption of differentially private training algorithms such as DP-SGD. However, a growing body of empirical work shows that DP-SGD often leads to suboptimal generalization performance, particularly on long-tailed data that contain a large number of rare or atypical samples. Despite these observations, a theoretical understanding of this phenomenon remains largely unexplored, and existing differential privacy analysis are difficult to extend to the nonconvex and nonsmooth neural networks commonly used in practice. In this work, we develop the first theoretical framework for analyzing DP-SGD on long-tailed data from a feature learning perspective. We show that the test error of DP-SGD-trained models on the long-tailed subpopulation is significantly larger than the overall test error over the entire dataset. Our analysis further characterizes the training dynamics of DP-SGD, demonstrating how gradient clipping and noise injection jointly adversely affect the model's ability to memorize informative but underrepresented samples. Finally, we validate our theoretical findings through extensive experiments on both synthetic and real-world datasets.