Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimization

TL;DR

This paper introduces a federated learning framework that combines sparsification and gradient perturbation to enhance privacy guarantees while maintaining communication efficiency, supported by rigorous analysis and experiments.

Contribution

It proposes a novel sparsification-amplified privacy method in federated learning, integrating acceleration techniques and rigorous Renyi DP analysis for improved privacy and efficiency.

Findings

Outperforms previous DP-FL methods in privacy guarantees.

Reduces communication rounds via acceleration techniques.

Achieves better privacy-utility trade-offs in experiments.

Abstract

Federated learning (FL) enables distributed agents to collaboratively learn a centralized model without sharing their raw data with each other. However, data locality does not provide sufficient privacy protection, and it is desirable to facilitate FL with rigorous differential privacy (DP) guarantee. Existing DP mechanisms would introduce random noise with magnitude proportional to the model size, which can be quite large in deep neural networks. In this paper, we propose a new FL framework with sparsification-amplified privacy. Our approach integrates random sparsification with gradient perturbation on each agent to amplify privacy guarantee. Since sparsification would increase the number of communication rounds required to achieve a certain target accuracy, which is unfavorable for DP guarantee, we further introduce acceleration techniques to help reduce the privacy cost. We rigorously analyze the convergence of our approach and utilize Renyi DP to tightly account the end-to-end DP guarantee. Extensive experiments on benchmark datasets validate that our approach outperforms previous differentially-private FL approaches in both privacy guarantee and communication efficiency.