Differential Privacy Meets Federated Learning under Communication Constraints

TL;DR

This paper explores the trade-offs between communication efficiency, training variance, and privacy in federated learning, providing theoretical and experimental insights for designing practical, privacy-aware systems under resource constraints.

Contribution

It offers a comprehensive analysis of how communication reduction techniques interact with differential privacy and data heterogeneity in federated learning.

Findings

Communication reduction techniques increase training variance.

Differential privacy impacts communication and variance trade-offs.

Insights guide practical design of privacy-preserving federated systems.

Abstract

The performance of federated learning systems is bottlenecked by communication costs and training variance. The communication overhead problem is usually addressed by three communication-reduction techniques, namely, model compression, partial device participation, and periodic aggregation, at the cost of increased training variance. Different from traditional distributed learning systems, federated learning suffers from data heterogeneity (since the devices sample their data from possibly different distributions), which induces additional variance among devices during training. Various variance-reduced training algorithms have been introduced to combat the effects of data heterogeneity, while they usually cost additional communication resources to deliver necessary control information. Additionally, data privacy remains a critical issue in FL, and thus there have been attempts at bringing Differential Privacy to this framework as a mediator between utility and privacy requirements. This paper investigates the trade-offs between communication costs and training variance under a resource-constrained federated system theoretically and experimentally, and how communication reduction techniques interplay in a differentially private setting. The results provide important insights into designing practical privacy-aware federated learning systems.