Dynamic Privacy For Distributed Machine Learning Over Network

TL;DR

This paper introduces two novel methods for ensuring differential privacy in distributed machine learning over networks, leveraging ADMM, and analyzes their performance and privacy-utility tradeoffs.

Contribution

It develops dual and primal variable perturbation techniques for dynamic differential privacy in distributed ERM problems, with performance analysis and privacy-utility tradeoff guidelines.

Findings

Dual variable perturbation outperforms primal perturbation.

Algorithms provide privacy guarantees under mild convexity conditions.

Numerical experiments validate privacy-utility tradeoffs.

Abstract

Privacy-preserving distributed machine learning becomes increasingly important due to the recent rapid growth of data. This paper focuses on a class of regularized empirical risk minimization (ERM) machine learning problems, and develops two methods to provide differential privacy to distributed learning algorithms over a network. We first decentralize the learning algorithm using the alternating direction method of multipliers (ADMM), and propose the methods of dual variable perturbation and primal variable perturbation to provide dynamic differential privacy. The two mechanisms lead to algorithms that can provide privacy guarantees under mild conditions of the convexity and differentiability of the loss function and the regularizer. We study the performance of the algorithms, and show that the dual variable perturbation outperforms its primal counterpart. To design an optimal privacy mechanisms, we analyze the fundamental tradeoff between privacy and accuracy, and provide guidelines to choose privacy parameters. Numerical experiments using customer information database are performed to corroborate the results on privacy and utility tradeoffs and design.