On the Tradeoff between Privacy Preservation and Byzantine-Robustness in Decentralized Learning

TL;DR

This paper explores the inherent tradeoff in decentralized learning between maintaining privacy through noise addition and ensuring robustness against Byzantine attacks, providing theoretical insights and practical guidelines.

Contribution

It introduces a unified analysis of privacy-robustness tradeoff in decentralized SGD, revealing how robust aggregation affects this balance.

Findings

Gaussian noise worsens learning error under Byzantine defenses

Robust aggregation rules' mixing abilities influence the privacy-robustness tradeoff

Theoretical guidelines for designing robust, privacy-preserving decentralized algorithms

Abstract

This paper jointly considers privacy preservation and Byzantine-robustness in decentralized learning. In a decentralized network, honest-but-curious agents faithfully follow the prescribed algorithm, but expect to infer their neighbors' private data from messages received during the learning process, while dishonest-and-Byzantine agents disobey the prescribed algorithm, and deliberately disseminate wrong messages to their neighbors so as to bias the learning process. For this novel setting, we investigate a generic privacy-preserving and Byzantine-robust decentralized stochastic gradient descent (SGD) framework, in which Gaussian noise is injected to preserve privacy and robust aggregation rules are adopted to counteract Byzantine attacks. We analyze its learning error and privacy guarantee, discovering an essential tradeoff between privacy preservation and Byzantine-robustness in decentralized learning -- the learning error caused by defending against Byzantine attacks is exacerbated by the Gaussian noise added to preserve privacy. For a class of state-of-the-art robust aggregation rules, we give unified analysis of the "mixing abilities". Building upon this analysis, we reveal how the "mixing abilities" affect the tradeoff between privacy preservation and Byzantine-robustness. The theoretical results provide guidelines for achieving a favorable tradeoff with proper design of robust aggregation rules. Numerical experiments are conducted and corroborate our theoretical findings.