Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails

TL;DR

This paper introduces Discriminative Clipping DPSGD, a novel method that distinguishes between body and tail gradients to improve privacy-preserving deep learning with heavy-tailed gradient distributions, reducing clipping loss and enhancing accuracy.

Contribution

The paper proposes a new discriminative clipping mechanism for DPSGD that effectively handles heavy-tailed gradients by separating body and tail components, improving gradient norm bounds.

Findings

Reduces empirical gradient norm from logarithmic to polylogarithmic scale.

Outperforms baseline methods by up to 9.72% in accuracy on real datasets.

Effectively manages heavy-tailed gradient distributions in deep learning.

Abstract

Differentially Private Stochastic Gradient Descent (DPSGD) is widely utilized to preserve training data privacy in deep learning, which first clips the gradients to a predefined norm and then injects calibrated noise into the training procedure. Existing DPSGD works typically assume the gradients follow sub-Gaussian distributions and design various clipping mechanisms to optimize training performance. However, recent studies have shown that the gradients in deep learning exhibit a heavy-tail phenomenon, that is, the tails of the gradient have infinite variance, which may lead to excessive clipping loss to the gradients with existing DPSGD mechanisms. To address this problem, we propose a novel approach, Discriminative Clipping~(DC)-DPSGD, with two key designs. First, we introduce a subspace identification technique to distinguish between body and tail gradients. Second, we present a discriminative clipping mechanism that applies different clipping thresholds for body and tail gradients to reduce the clipping loss. Under the non-convex condition, \ourtech{} reduces the empirical gradient norm from {${\mathbb{O}\left(\log^{\max(0,\theta-1)}(T/\delta)\log^{2\theta}(\sqrt{T})\right)}$} to {${\mathbb{O}\left(\log(\sqrt{T})\right)}$} with heavy-tailed index $\theta\geq 1/2$, iterations $T$, and arbitrary probability $\delta$. Extensive experiments on four real-world datasets demonstrate that our approach outperforms three baselines by up to 9.72\% in terms of accuracy.