Dyn-D$^2$P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guarantee

TL;DR

This paper introduces Dyn-D$^2$P, a dynamic differential privacy method for decentralized learning that adjusts noise and clipping bounds based on convergence, improving accuracy while maintaining privacy guarantees.

Contribution

It proposes a novel dynamic noise and clipping strategy for decentralized DP learning, with a provable utility bound and improved accuracy over fixed-noise methods.

Findings

Dyn-D$^2$P outperforms fixed-noise methods in accuracy under strong privacy.

The utility bound explicitly relates to network parameters and scales with 1/√n.

First utility analysis for dynamic gradient clipping in decentralized DP optimization.

Abstract

Most existing decentralized learning methods with differential privacy (DP) guarantee rely on constant gradient clipping bounds and fixed-level DP Gaussian noises for each node throughout the training process, leading to a significant accuracy degradation compared to non-private counterparts. In this paper, we propose a new Dynamic Differentially Private Decentralized learning approach (termed Dyn-D$^2$P) tailored for general time-varying directed networks. Leveraging the Gaussian DP (GDP) framework for privacy accounting, Dyn-D$^2$P dynamically adjusts gradient clipping bounds and noise levels based on gradient convergence. This proposed dynamic noise strategy enables us to enhance model accuracy while preserving the total privacy budget. Extensive experiments on benchmark datasets demonstrate the superiority of Dyn-D$^2$P over its counterparts employing fixed-level noises, especially under strong privacy guarantees. Furthermore, we provide a provable utility bound for Dyn-D$^2$P that establishes an explicit dependency on network-related parameters, with a scaling factor of $1/\sqrt{n}$ in terms of the number of nodes $n$ up to a bias error term induced by gradient clipping. To our knowledge, this is the first model utility analysis for differentially private decentralized non-convex optimization with dynamic gradient clipping bounds and noise levels.