To Shuffle or not to Shuffle: Auditing DP-SGD with Shuffling

TL;DR

This paper introduces novel auditing procedures to evaluate the privacy guarantees of DP-SGD with shuffling, revealing that current models often overestimate their privacy protection, with leakage up to 4-10 times higher than reported.

Contribution

The paper develops DP-auditing methods to accurately measure privacy leakage in shuffling-based DP-SGD, exposing overestimations in existing privacy guarantees and analyzing variations in shuffling procedures.

Findings

DP models overestimate privacy guarantees by up to 4 times.

The gap between theoretical and actual privacy leakage varies with parameters.

Certain shuffling variations increase privacy leakage up to 10 times.

Abstract

The Differentially Private Stochastic Gradient Descent (DP-SGD) algorithm supports the training of machine learning (ML) models with formal Differential Privacy (DP) guarantees. Traditionally, DP-SGD processes training data in batches using Poisson subsampling to select each batch at every iteration. More recently, shuffling has become a common alternative due to its better compatibility and lower computational overhead. However, computing tight theoretical DP guarantees under shuffling remains an open problem. As a result, models trained with shuffling are often evaluated as if Poisson subsampling were used, which might result in incorrect privacy guarantees. This raises a compelling research question: can we verify whether there are gaps between the theoretical DP guarantees reported by state-of-the-art models using shuffling and their actual leakage? To do so, we define novel DP-auditing procedures to analyze DP-SGD with shuffling and measure their ability to tightly estimate privacy leakage vis-\`a-vis batch sizes, privacy budgets, and threat models. Overall, we demonstrate that DP models trained using this approach have considerably overestimated their privacy guarantees (by up to 4 times). However, we also find that the gap between the theoretical Poisson DP guarantees and the actual privacy leakage from shuffling is not uniform across all parameter settings and threat models. Finally, we study two common variations of the shuffling procedure that result in even further privacy leakage (up to 10 times). Overall, our work highlights the risk of using shuffling instead of Poisson subsampling in the absence of rigorous analysis methods.