DP-ADMM: ADMM-based Distributed Learning with Differential Privacy

TL;DR

This paper introduces DP-ADMM, a novel differentially private distributed learning algorithm based on ADMM, which improves utility under high privacy guarantees and applies to a broad class of problems.

Contribution

It proposes a new DP-ADMM algorithm combining an approximate augmented Lagrangian with Gaussian noise, providing explicit convergence and utility guarantees for general objectives.

Findings

Achieves high utility under strong privacy guarantees.

Proven convergence for a wide class of problems.

Demonstrates good accuracy and convergence in experiments.

Abstract

Alternating Direction Method of Multipliers (ADMM) is a widely used tool for machine learning in distributed settings, where a machine learning model is trained over distributed data sources through an interactive process of local computation and message passing. Such an iterative process could cause privacy concerns of data owners. The goal of this paper is to provide differential privacy for ADMM-based distributed machine learning. Prior approaches on differentially private ADMM exhibit low utility under high privacy guarantee and often assume the objective functions of the learning problems to be smooth and strongly convex. To address these concerns, we propose a novel differentially private ADMM-based distributed learning algorithm called DP-ADMM, which combines an approximate augmented Lagrangian function with time-varying Gaussian noise addition in the iterative process to achieve higher utility for general objective functions under the same differential privacy guarantee. We also apply the moments accountant method to bound the end-to-end privacy loss. The theoretical analysis shows that DP-ADMM can be applied to a wider class of distributed learning problems, is provably convergent, and offers an explicit utility-privacy tradeoff. To our knowledge, this is the first paper to provide explicit convergence and utility properties for differentially private ADMM-based distributed learning algorithms. The evaluation results demonstrate that our approach can achieve good convergence and model accuracy under high end-to-end differential privacy guarantee.