Differentially Private Statistical Inference through $\beta$-Divergence One Posterior Sampling

TL;DR

This paper introduces $eta$D-Bayes, a novel differentially private Bayesian posterior sampling method based on $eta$-divergence, enabling more precise private inference for complex models without altering the data generation process.

Contribution

It proposes a generalised posterior sampling scheme that improves private inference accuracy and applicability to complex models like neural networks without changing the underlying model.

Findings

$eta$D-Bayes achieves more precise private estimates at the same privacy level.

It enables differentially private estimation for complex classifiers and neural networks.

The method does not require modifications to the original data generative model.

Abstract

Differential privacy guarantees allow the results of a statistical analysis involving sensitive data to be released without compromising the privacy of any individual taking part. Achieving such guarantees generally requires the injection of noise, either directly into parameter estimates or into the estimation process. Instead of artificially introducing perturbations, sampling from Bayesian posterior distributions has been shown to be a special case of the exponential mechanism, producing consistent, and efficient private estimates without altering the data generative process. The application of current approaches has, however, been limited by their strong bounding assumptions which do not hold for basic models, such as simple linear regressors. To ameliorate this, we propose $\beta$D-Bayes, a posterior sampling scheme from a generalised posterior targeting the minimisation of the $\beta$-divergence between the model and the data generating process. This provides private estimation that is generally applicable without requiring changes to the underlying model and consistently learns the data generating parameter. We show that $\beta$D-Bayes produces more precise inference estimation for the same privacy guarantees, and further facilitates differentially private estimation via posterior sampling for complex classifiers and continuous regression models such as neural networks for the first time.