Continual Release Moment Estimation with Differential Privacy

TL;DR

This paper introduces Joint Moment Estimation (JME), a differentially private method that accurately estimates data moments over time with less noise, improving privacy-preserving data analysis and model training.

Contribution

JME is a novel approach that uses the matrix mechanism and joint sensitivity analysis to estimate moments privately with reduced noise, enhancing accuracy in continual privacy settings.

Findings

JME reduces noise in moment estimation compared to naive methods.

JME achieves accurate mean and covariance estimation for Gaussian models.

JME improves model training with DP-Adam on CIFAR-10.

Abstract

We propose Joint Moment Estimation (JME), a method for continually and privately estimating both the first and second moments of data with reduced noise compared to naive approaches. JME uses the matrix mechanism and a joint sensitivity analysis to allow the second moment estimation with no additional privacy cost, thereby improving accuracy while maintaining privacy. We demonstrate JME's effectiveness in two applications: estimating the running mean and covariance matrix for Gaussian density estimation, and model training with DP-Adam on CIFAR-10.