Generalizing DP-SGD with Shuffling and Batch Clipping

TL;DR

This paper introduces a flexible differential privacy framework that extends DP-SGD to support various optimizers, batch clipping, and shuffling, providing new privacy guarantees and analysis techniques.

Contribution

It proposes a general DP algorithmic framework enabling any first-order optimizer with batch clipping and shuffling, along with a novel privacy analysis method.

Findings

Achieves a $ ext{sqrt}(g E)$ DP dependency for groups of size g over E epochs.

Provides closed-form expressions for privacy analysis.

Introduces a new proof technique for group privacy in differential privacy.

Abstract

Classical differential private DP-SGD implements individual clipping with random subsampling, which forces a mini-batch SGD approach. We provide a general differential private algorithmic framework that goes beyond DP-SGD and allows any possible first order optimizers (e.g., classical SGD and momentum based SGD approaches) in combination with batch clipping, which clips an aggregate of computed gradients rather than summing clipped gradients (as is done in individual clipping). The framework also admits sampling techniques beyond random subsampling such as shuffling. Our DP analysis follows the $f$-DP approach and introduces a new proof technique which allows us to derive simple closed form expressions and to also analyse group privacy. In particular, for $E$ epochs work and groups of size $g$, we show a $\sqrt{g E}$ DP dependency for batch clipping with shuffling.