Lightweight Federated Learning with Differential Privacy and Straggler Resilience

TL;DR

LightDP-FL is a lightweight federated learning scheme that provides differential privacy, resilience to stragglers, and maintains high accuracy with minimal overhead, outperforming baseline methods in convergence speed and robustness.

Contribution

The paper introduces LightDP-FL, a novel federated learning approach that combines individual and pairwise noise for differential privacy with low overhead and straggler resilience.

Findings

Faster convergence compared to baseline methods.

Stronger resilience to stragglers.

Maintains high accuracy under differential privacy constraints.

Abstract

Federated learning (FL) enables collaborative model training through model parameter exchanges instead of raw data. To avoid potential inference attacks from exchanged parameters, differential privacy (DP) offers rigorous guarantee against various attacks. However, conventional methods of ensuring DP by adding local noise alone often result in low training accuracy. Combining secure multi-party computation (SMPC) with DP, while improving the accuracy, incurs high communication and computation overheads as well as straggler vulnerability, in either client-to-server or client-to-client links. In this paper, we propose LightDP-FL, a novel lightweight scheme that ensures provable DP against untrusted peers and server, while maintaining straggler resilience, low overheads and high training accuracy. Our scheme incorporates both individual and pairwise noise into each client's parameter, which can be implemented with minimal overheads. Given the uncertain straggler and colluder sets, we utilize the upper bound on the numbers of stragglers and colluders to prove sufficient noise variance conditions to ensure DP in the worst case. Moreover, we optimize the expected convergence bound to ensure accuracy performance by flexibly controlling the noise variances. Using the CIFAR-10 dataset, our experimental results demonstrate that LightDP-FL achieves faster convergence and stronger straggler resilience compared to baseline methods of the same DP level.