Statistical Inference for Differentially Private Stochastic Gradient Descent

TL;DR

This paper develops statistical inference methods for DP-SGD, providing asymptotic analysis and confidence intervals that account for privacy, sampling, and statistical variability, ensuring valid uncertainty quantification in private machine learning.

Contribution

It extends asymptotic theory to DP-SGD with randomized subsampling and proposes two practical methods for valid confidence intervals under differential privacy.

Findings

Confidence intervals achieve nominal coverage rates.

Proposed methods effectively incorporate privacy and sampling variability.

Asymptotic variance decomposes into statistical, sampling, and privacy components.

Abstract

Privacy preservation in machine learning, particularly through Differentially Private Stochastic Gradient Descent (DP-SGD), is critical for sensitive data analysis. However, existing statistical inference methods for SGD predominantly focus on cyclic subsampling, while DP-SGD requires randomized subsampling. This paper first bridges this gap by establishing the asymptotic properties of SGD under the randomized rule and extending these results to DP-SGD. For the output of DP-SGD, we show that the asymptotic variance decomposes into statistical, sampling, and privacy-induced components. Two methods are proposed for constructing valid confidence intervals: the plug-in method and the random scaling method. We also perform extensive numerical analysis, which shows that the proposed confidence intervals achieve nominal coverage rates while maintaining privacy.