Differential Privacy Has Disparate Impact on Model Accuracy

TL;DR

Differential privacy in neural network training disproportionately reduces accuracy for underrepresented groups, exacerbating existing biases and unfairness across various tasks and models.

Contribution

This paper reveals that differential privacy mechanisms worsen model fairness by disproportionately impacting underrepresented groups, a previously underexplored consequence.

Findings

DP models show larger accuracy drops for underrepresented groups

The fairness gap increases when applying DP compared to non-DP models

Gradient clipping and noise addition disproportionately affect complex and underrepresented subgroups

Abstract

Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.