Differentially Private Block-wise Gradient Shuffle for Deep Learning

TL;DR

This paper introduces DP-BloGS, a novel differentially private deep learning algorithm that uses gradient shuffling to improve privacy and training efficiency, demonstrating enhanced resistance to data extraction.

Contribution

DP-BloGS shifts from traditional noise addition to a probabilistic shuffling approach, achieving near non-private training speeds with strong privacy guarantees.

Findings

DP-BloGS achieves training times close to non-private methods.

DP-BloGS maintains similar privacy and utility as DP-SGD.

DP-BloGS is more resistant to data extraction attacks.

Abstract

Traditional Differentially Private Stochastic Gradient Descent (DP-SGD) introduces statistical noise on top of gradients drawn from a Gaussian distribution to ensure privacy. This paper introduces the novel Differentially Private Block-wise Gradient Shuffle (DP-BloGS) algorithm for deep learning. BloGS builds off of existing private deep learning literature, but makes a definitive shift by taking a probabilistic approach to gradient noise introduction through shuffling modeled after information theoretic privacy analyses. The theoretical results presented in this paper show that the combination of shuffling, parameter-specific block size selection, batch layer clipping, and gradient accumulation allows DP-BloGS to achieve training times close to that of non-private training while maintaining similar privacy and utility guarantees to DP-SGD. DP-BloGS is found to be significantly more resistant to data extraction attempts than DP-SGD. The theoretical results are validated by the experimental findings.