On Connecting Stochastic Gradient MCMC and Differential Privacy

TL;DR

This paper demonstrates that stochastic gradient MCMC algorithms inherently satisfy differential privacy under certain conditions, enabling private Bayesian learning with strong theoretical guarantees and practical effectiveness.

Contribution

It establishes a theoretical connection between SG-MCMC and differential privacy, showing that standard SG-MCMC can achieve privacy without modifications.

Findings

SG-MCMC satisfies differential privacy with appropriate step sizes

Standard SG-MCMC achieves state-of-the-art privacy-utility trade-offs

Experimental results confirm theoretical analysis

Abstract

Significant success has been realized recently on applying machine learning to real-world applications. There have also been corresponding concerns on the privacy of training data, which relates to data security and confidentiality issues. Differential privacy provides a principled and rigorous privacy guarantee on machine learning models. While it is common to design a model satisfying a required differential-privacy property by injecting noise, it is generally hard to balance the trade-off between privacy and utility. We show that stochastic gradient Markov chain Monte Carlo (SG-MCMC) -- a class of scalable Bayesian posterior sampling algorithms proposed recently -- satisfies strong differential privacy with carefully chosen step sizes. We develop theory on the performance of the proposed differentially-private SG-MCMC method. We conduct experiments to support our analysis and show that a standard SG-MCMC sampler without any modification (under a default setting) can reach state-of-the-art performance in terms of both privacy and utility on Bayesian learning.