Training with Differential Privacy: A Gradient-Preserving Noise Reduction Approach with Provable Security

TL;DR

This paper introduces GReDP, a novel differentially private training method that reduces noise by operating in the frequency domain, maintaining model utility while ensuring provable security.

Contribution

GReDP is a new approach that computes gradients in the frequency domain to lower noise levels, outperforming existing methods like DPSGD in privacy-preserving deep learning.

Findings

GReDP requires only half the noise scale of DPSGD.

GReDP maintains higher model utility across various models and training settings.

Theoretical and empirical analyses confirm GReDP's effectiveness and security.

Abstract

Deep learning models have been extensively adopted in various regions due to their ability to represent hierarchical features, which highly rely on the training set and procedures. Thus, protecting the training process and deep learning algorithms is paramount in privacy preservation. Although Differential Privacy (DP) as a powerful cryptographic primitive has achieved satisfying results in deep learning training, the existing schemes still fall short in preserving model utility, i.e., they either invoke a high noise scale or inevitably harm the original gradients. To address the above issues, in this paper, we present a more robust and provably secure approach for differentially private training called GReDP. Specifically, we compute the model gradients in the frequency domain and adopt a new approach to reduce the noise level. Unlike previous work, our GReDP only requires half of the noise scale compared to DPSGD [1] while keeping all the gradient information intact. We present a detailed analysis of our method both theoretically and empirically. The experimental results show that our GReDP works consistently better than the baselines on all models and training settings.