Too Good to be True? Turn Any Model Differentially Private With DP-Weights

TL;DR

This paper introduces a novel post-training method called DP-Weights that applies differential privacy noise to model weights after training, enabling privacy-utility trade-offs without retraining.

Contribution

The study presents the first formal proof and empirical validation of post-training noise addition to achieve differential privacy, reducing training time and increasing flexibility.

Findings

Post-training noise application achieves comparable privacy guarantees to traditional methods.

The approach maintains model utility with minimal performance loss.

Significant time savings by avoiding retraining from scratch.

Abstract

Imagine training a machine learning model with Differentially Private Stochastic Gradient Descent (DP-SGD), only to discover post-training that the noise level was either too high, crippling your model's utility, or too low, compromising privacy. The dreaded realization hits: you must start the lengthy training process from scratch. But what if you could avoid this retraining nightmare? In this study, we introduce a groundbreaking approach (to our knowledge) that applies differential privacy noise to the model's weights after training. We offer a comprehensive mathematical proof for this novel approach's privacy bounds, use formal methods to validate its privacy guarantees, and empirically evaluate its effectiveness using membership inference attacks and performance evaluations. This method allows for a single training run, followed by post-hoc noise adjustments to achieve optimal privacy-utility trade-offs. We compare this novel fine-tuned model (DP-Weights model) to a traditional DP-SGD model, demonstrating that our approach yields statistically similar performance and privacy guarantees. Our results validate the efficacy of post-training noise application, promising significant time savings and flexibility in fine-tuning differential privacy parameters, making it a practical alternative for deploying differentially private models in real-world scenarios.