Enforcing Privacy in Distributed Learning with Performance Guarantees

TL;DR

This paper investigates privacy-preserving distributed learning, revealing limitations of common methods and proposing graph-structured alternatives that enhance performance without sacrificing privacy, even without assuming bounded gradients.

Contribution

It introduces novel graph-homomorphic privacy schemes that outperform additive noise methods and removes the bounded gradient assumption in differential privacy analysis.

Findings

Graph-homomorphic schemes improve performance over additive noise.

The proposed methods guarantee privacy without bounded gradient assumptions.

Simulations validate theoretical improvements.

Abstract

We study the privatization of distributed learning and optimization strategies. We focus on differential privacy schemes and study their effect on performance. We show that the popular additive random perturbation scheme degrades performance because it is not well-tuned to the graph structure. For this reason, we exploit two alternative graph-homomorphic constructions and show that they improve performance while guaranteeing privacy. Moreover, contrary to most earlier studies, the gradient of the risks is not assumed to be bounded (a condition that rarely holds in practice; e.g., quadratic risk). We avoid this condition and still devise a differentially private scheme with high probability. We examine optimization and learning scenarios and illustrate the theoretical findings through simulations.