Differentially Private Hamiltonian Monte Carlo

TL;DR

This paper introduces a differentially private version of Hamiltonian Monte Carlo that maintains convergence and ergodicity by incorporating the Metropolis-Hastings test and noisy gradients, improving privacy-preserving Bayesian inference.

Contribution

It develops a novel DP-HMC algorithm using the MH test and gradient noise, ensuring convergence and better performance compared to existing DP MCMC methods.

Findings

DP-HMC converges to the correct distribution.

DP-HMC outperforms or matches existing DP MCMC algorithms.

DP-HMC demonstrates consistent performance across experiments.

Abstract

Markov chain Monte Carlo (MCMC) algorithms have long been the main workhorses of Bayesian inference. Among them, Hamiltonian Monte Carlo (HMC) has recently become very popular due to its efficiency resulting from effective use of the gradients of the target distribution. In privacy-preserving machine learning, differential privacy (DP) has become the gold standard in ensuring that the privacy of data subjects is not violated. Existing DP MCMC algorithms either use random-walk proposals, or do not use the Metropolis--Hastings (MH) acceptance test to ensure convergence without decreasing their step size to zero. We present a DP variant of HMC using the MH acceptance test that builds on a recently proposed DP MCMC algorithm called the penalty algorithm, and adds noise to the gradient evaluations of HMC. We prove that the resulting algorithm converges to the correct distribution, and is ergodic. We compare DP-HMC with the existing penalty, DP-SGLD and DP-SGNHT algorithms, and find that DP-HMC has better or equal performance than the penalty algorithm, and performs more consistently than DP-SGLD or DP-SGNHT.