Differential Privacy in Two-Layer Networks: How DP-SGD Harms Fairness and Robustness

TL;DR

This paper analyzes how differential privacy via DP-SGD affects feature learning, fairness, and robustness in two-layer neural networks, revealing that privacy noise causes disparities and vulnerabilities.

Contribution

It introduces a feature-centric framework for analyzing DP-SGD in neural networks, linking privacy noise to fairness and robustness issues with theoretical bounds.

Findings

Imbalanced feature-to-noise ratios cause disparate impact across classes.

Noise impacts long-tailed data more severely within the same class.

Privacy noise increases vulnerability to adversarial attacks.

Abstract

Differentially private learning is essential for training models on sensitive data, but empirical studies consistently show that it can degrade performance, introduce fairness issues like disparate impact, and reduce adversarial robustness. The theoretical underpinnings of these phenomena in modern, non-convex neural networks remain largely unexplored. This paper introduces a unified feature-centric framework to analyze the feature learning dynamics of differentially private stochastic gradient descent (DP-SGD) in two-layer ReLU convolutional neural networks. Our analysis establishes test loss bounds governed by a crucial metric: the feature-to-noise ratio (FNR). We demonstrate that the noise required for privacy leads to suboptimal feature learning, and specifically show that: 1) imbalanced FNRs across classes and subpopulations cause disparate impact; 2) even in the same class, noise has a greater negative impact on semantically long-tailed data; and 3) noise injection exacerbates vulnerability to adversarial attacks. Furthermore, our analysis reveals that the popular paradigm of public pre-training and private fine-tuning does not guarantee improvement, particularly under significant feature distribution shifts between datasets. Experiments on synthetic and real-world data corroborate our theoretical findings.