Universally Harmonizing Differential Privacy Mechanisms for Federated Learning: Boosting Accuracy and Convergence

TL;DR

This paper introduces UDP-FL, a universal framework for differential privacy in federated learning that improves accuracy and convergence by harmonizing various noise mechanisms with DP-SGD, backed by theoretical analysis and extensive experiments.

Contribution

The paper presents the first universal DP-FL framework that harmonizes any randomization mechanism with DP-SGD, enhancing privacy-accuracy tradeoff and convergence in federated learning.

Findings

UDP-FL significantly improves model accuracy and convergence.

UDP-FL outperforms state-of-the-art methods in privacy and performance.

UDP-FL is resilient against inference attacks.

Abstract

Differentially private federated learning (DP-FL) is a promising technique for collaborative model training while ensuring provable privacy for clients. However, optimizing the tradeoff between privacy and accuracy remains a critical challenge. To our best knowledge, we propose the first DP-FL framework (namely UDP-FL), which universally harmonizes any randomization mechanism (e.g., an optimal one) with the Gaussian Moments Accountant (viz. DP-SGD) to significantly boost accuracy and convergence. Specifically, UDP-FL demonstrates enhanced model performance by mitigating the reliance on Gaussian noise. The key mediator variable in this transformation is the R\'enyi Differential Privacy notion, which is carefully used to harmonize privacy budgets. We also propose an innovative method to theoretically analyze the convergence for DP-FL (including our UDP-FL ) based on mode connectivity analysis. Moreover, we evaluate our UDP-FL through extensive experiments benchmarked against state-of-the-art (SOTA) methods, demonstrating superior performance on both privacy guarantees and model performance. Notably, UDP-FL exhibits substantial resilience against different inference attacks, indicating a significant advance in safeguarding sensitive data in federated learning environments.