Enhancing DPSGD via Per-Sample Momentum and Low-Pass Filtering

TL;DR

This paper introduces DP-PMLF, a novel method that combines per-sample momentum and low-pass filtering to reduce both noise and bias in DPSGD, leading to improved model accuracy under privacy constraints.

Contribution

The paper proposes DP-PMLF, a new approach integrating momentum and filtering to address both noise and bias in DPSGD, with theoretical analysis and empirical validation.

Findings

Significantly improves privacy-utility trade-off in DPSGD

Reduces sampling variance and high-frequency noise effectively

Achieves better convergence rates under differential privacy constraints

Abstract

Differentially Private Stochastic Gradient Descent (DPSGD) is widely used to train deep neural networks with formal privacy guarantees. However, the addition of differential privacy (DP) often degrades model accuracy by introducing both noise and bias. Existing techniques typically address only one of these issues, as reducing DP noise can exacerbate clipping bias and vice-versa. In this paper, we propose a novel method, \emph{DP-PMLF}, which integrates per-sample momentum with a low-pass filtering strategy to simultaneously mitigate DP noise and clipping bias. Our approach uses per-sample momentum to smooth gradient estimates prior to clipping, thereby reducing sampling variance. It further employs a post-processing low-pass filter to attenuate high-frequency DP noise without consuming additional privacy budget. We provide a theoretical analysis demonstrating an improved convergence rate under rigorous DP guarantees, and our empirical evaluations reveal that DP-PMLF significantly enhances the privacy-utility trade-off compared to several state-of-the-art DPSGD variants.