Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight

TL;DR

This paper introduces DP-PSASC, an improved DP-SGD method using non-monotonous adaptive gradient scaling and momentum to better preserve privacy and enhance model accuracy, especially for small gradients.

Contribution

It proposes a novel gradient scaling technique and integrates momentum into DP-PSASC, advancing privacy-preserving deep learning with improved convergence and performance.

Findings

DP-PSASC outperforms traditional DP-SGD in accuracy.

The new method effectively handles small gradients during training.

The approach maintains privacy guarantees while improving utility.

Abstract

In the domain of deep learning, the challenge of protecting sensitive data while maintaining model utility is significant. Traditional Differential Privacy (DP) techniques such as Differentially Private Stochastic Gradient Descent (DP-SGD) typically employ strategies like direct or per-sample adaptive gradient clipping. These methods, however, compromise model accuracy due to their critical influence on gradient handling, particularly neglecting the significant contribution of small gradients during later training stages. In this paper, we introduce an enhanced version of DP-SGD, named Differentially Private Per-sample Adaptive Scaling Clipping (DP-PSASC). This approach replaces traditional clipping with non-monotonous adaptive gradient scaling, which alleviates the need for intensive threshold setting and rectifies the disproportionate weighting of smaller gradients. Our contribution is twofold. First, we develop a novel gradient scaling technique that effectively assigns proper weights to gradients, particularly small ones, thus improving learning under differential privacy. Second, we integrate a momentum-based method into DP-PSASC to reduce bias from stochastic sampling, enhancing convergence rates. Our theoretical and empirical analyses confirm that DP-PSASC preserves privacy and delivers superior performance across diverse datasets, setting new standards for privacy-sensitive applications.