Tractable MCMC for Private Learning with Pure and Gaussian Differential Privacy

TL;DR

This paper introduces a new MCMC-based algorithm that ensures pure differential privacy in Bayesian sampling, overcoming the privacy-accuracy trade-off caused by approximate sampling methods.

Contribution

The paper proposes ASAP, an MCMC perturbation method that guarantees pure DP, and demonstrates its convergence and efficiency in DP-ERM problems with convex losses.

Findings

ASAP achieves pure differential privacy with controlled Wasserstein-infinity distance.

The algorithm converges in Wasserstein-infinity distance.

It provides the first nearly linear-time solution for DP-ERM with optimal rates.

Abstract

Posterior sampling, i.e., exponential mechanism to sample from the posterior distribution, provides $\varepsilon$-pure differential privacy (DP) guarantees and does not suffer from potentially unbounded privacy breach introduced by $(\varepsilon,\delta)$-approximate DP. In practice, however, one needs to apply approximate sampling methods such as Markov chain Monte Carlo (MCMC), thus re-introducing the unappealing $\delta$-approximation error into the privacy guarantees. To bridge this gap, we propose the Approximate SAample Perturbation (abbr. ASAP) algorithm which perturbs an MCMC sample with noise proportional to its Wasserstein-infinity ($W_\infty$) distance from a reference distribution that satisfies pure DP or pure Gaussian DP (i.e., $\delta=0$). We then leverage a Metropolis-Hastings algorithm to generate the sample and prove that the algorithm converges in $W_\infty$ distance. We show that by combining our new techniques with a localization step, we obtain the first nearly linear-time algorithm that achieves the optimal rates in the DP-ERM problem with strongly convex and smooth losses.