Differentially Private ADMM Algorithms for Machine Learning

TL;DR

This paper introduces differentially private ADMM algorithms for machine learning that balance privacy and utility, providing theoretical guarantees and demonstrating improved convergence and utility through experiments.

Contribution

The paper proposes the first differentially private ADMM algorithms with performance guarantees and acceleration techniques for convex machine learning problems.

Findings

DP-ADMM guarantees $(,)$-DP with performance bounds.

DP-AccADMM accelerates convergence and improves utility.

Algorithms demonstrate favorable privacy-utility tradeoffs in experiments.

Abstract

In this paper, we study efficient differentially private alternating direction methods of multipliers (ADMM) via gradient perturbation for many machine learning problems. For smooth convex loss functions with (non)-smooth regularization, we propose the first differentially private ADMM (DP-ADMM) algorithm with performance guarantee of $(\epsilon,\delta)$-differential privacy ($(\epsilon,\delta)$-DP). From the viewpoint of theoretical analysis, we use the Gaussian mechanism and the conversion relationship between R\'enyi Differential Privacy (RDP) and DP to perform a comprehensive privacy analysis for our algorithm. Then we establish a new criterion to prove the convergence of the proposed algorithms including DP-ADMM. We also give the utility analysis of our DP-ADMM. Moreover, we propose an accelerated DP-ADMM (DP-AccADMM) with the Nesterov's acceleration technique. Finally, we conduct numerical experiments on many real-world datasets to show the privacy-utility tradeoff of the two proposed algorithms, and all the comparative analysis shows that DP-AccADMM converges faster and has a better utility than DP-ADMM, when the privacy budget $\epsilon$ is larger than a threshold.