DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction

TL;DR

This paper introduces DOPPLER, a novel DP optimizer that uses low-pass filtering in the frequency domain to reduce privacy noise impact, significantly improving model accuracy while maintaining privacy guarantees.

Contribution

The paper proposes a new frequency domain approach with low-pass filtering for DP optimizers, enhancing performance without compromising privacy.

Findings

DOPPLER improves test accuracy by 3%-10% over traditional DP optimizers.

The low-pass filter effectively reduces DP noise impact in training.

The approach maintains rigorous privacy guarantees while enhancing model utility.

Abstract

Privacy is a growing concern in modern deep-learning systems and applications. Differentially private (DP) training prevents the leakage of sensitive information in the collected training data from the trained machine learning models. DP optimizers, including DP stochastic gradient descent (DPSGD) and its variants, privatize the training procedure by gradient clipping and DP noise injection. However, in practice, DP models trained using DPSGD and its variants often suffer from significant model performance degradation. Such degradation prevents the application of DP optimization in many key tasks, such as foundation model pretraining. In this paper, we provide a novel signal processing perspective to the design and analysis of DP optimizers. We show that a ``frequency domain'' operation called low-pass filtering can be used to effectively reduce the impact of DP noise. More specifically, by defining the ``frequency domain'' for both the gradient and differential privacy (DP) noise, we have developed a new component, called DOPPLER. This component is designed for DP algorithms and works by effectively amplifying the gradient while suppressing DP noise within this frequency domain. As a result, it maintains privacy guarantees and enhances the quality of the DP-protected model. Our experiments show that the proposed DP optimizers with a low-pass filter outperform their counterparts without the filter by 3%-10% in test accuracy on various models and datasets. Both theoretical and practical evidence suggest that the DOPPLER is effective in closing the gap between DP and non-DP training.