Noise-Aware Differentially Private Variational Inference

TL;DR

This paper introduces a noise-aware variational inference method that maintains differential privacy guarantees while effectively handling high-dimensional and complex models, improving accuracy and calibration in privacy-preserving Bayesian inference.

Contribution

It proposes a novel noise-aware variational inference approach applicable to high-dimensional, non-conjugate models, with an improved evaluation method for privacy-preserving posteriors.

Findings

Comparable performance to existing methods in simple models

Accurate coverage in high-dimensional Bayesian linear regression

Well-calibrated predictions on UCI Adult dataset

Abstract

Differential privacy (DP) provides robust privacy guarantees for statistical inference, but this can lead to unreliable results and biases in downstream applications. While several noise-aware approaches have been proposed which integrate DP perturbation into the inference, they are limited to specific types of simple probabilistic models. In this work, we propose a novel method for noise-aware approximate Bayesian inference based on stochastic gradient variational inference which can also be applied to high-dimensional and non-conjugate models. We also propose a more accurate evaluation method for noise-aware posteriors. Empirically, our inference method has similar performance to existing methods in the domain where they are applicable. Outside this domain, we obtain accurate coverages on high-dimensional Bayesian linear regression and well-calibrated predictive probabilities on Bayesian logistic regression with the UCI Adult dataset.