An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Losses

TL;DR

This paper provides a rigorous analysis of the privacy and utility trade-offs in differentially private SGD for non-convex, smooth loss functions within bounded domains, improving theoretical bounds and practical understanding.

Contribution

It introduces a new privacy characterization for DPSGD with non-convex, smooth losses in bounded domains, removing convexity assumptions and analyzing convergence of privacy loss.

Findings

Privacy loss converges over iterations without convexity.

Smaller domain bounds enhance privacy and utility.

Theoretical bounds on privacy-utility trade-off are established.

Abstract

Differentially Private Stochastic Gradient Descent (DPSGD) is widely used to protect sensitive data during the training of machine learning models, but its privacy guarantee often comes at a large cost of model performance due to the lack of tight theoretical bounds quantifying privacy loss. While recent efforts have achieved more accurate privacy guarantees, they still impose some assumptions prohibited from practical applications, such as convexity and complex parameter requirements, and rarely investigate in-depth the impact of privacy mechanisms on the model's utility. In this paper, we provide a rigorous privacy characterization for DPSGD with general L-smooth and non-convex loss functions, revealing converged privacy loss with iteration in bounded-domain cases. Specifically, we track the privacy loss over multiple iterations, leveraging the noisy smooth-reduction property, and further establish comprehensive convergence analysis in different scenarios. In particular, we show that for DPSGD with a bounded domain, (i) the privacy loss can still converge without the convexity assumption, (ii) a smaller bounded diameter can improve both privacy and utility simultaneously under certain conditions, and (iii) the attainable big-O order of the privacy utility trade-off for DPSGD with gradient clipping (DPSGD-GC) and for DPSGD-GC with bounded domain (DPSGD-DC) and mu-strongly convex population risk function, respectively. Experiments via membership inference attack (MIA) in a practical setting validate insights gained from the theoretical results.