Generalizing Differentially Private Decentralized Deep Learning with Multi-Agent Consensus

TL;DR

This paper introduces a framework for differentially private decentralized deep learning that guarantees convergence and privacy, achieving near-centralized accuracy while resisting inference attacks.

Contribution

It embeds differential privacy into decentralized deep learning algorithms, providing convergence guarantees and analyzing the impact of privacy and network structure on accuracy.

Findings

Achieves near-centralized accuracy in decentralized learning.

Ensures individual data privacy against inference attacks.

Identifies invariance of accuracy to communication graph structure beyond a threshold.

Abstract

Cooperative decentralized learning relies on direct information exchange between communicating agents, each with access to locally available datasets. The goal is to agree on model parameters that are optimal over all data. However, sharing parameters with untrustworthy neighbors can incur privacy risks by leaking exploitable information. To enable trustworthy cooperative learning, we propose a framework that embeds differential privacy into decentralized deep learning and secures each agent's local dataset during and after cooperative training. We prove convergence guarantees for algorithms derived from this framework and demonstrate its practical utility when applied to subgradient and ADMM decentralized approaches, finding accuracies approaching the centralized baseline while ensuring individual data samples are resilient to inference attacks. Furthermore, we study the relationships between accuracy, privacy budget, and networks' graph properties on collaborative classification tasks, discovering a useful invariance to the communication graph structure beyond a threshold.