Differentially Private Clustered Federated Learning

TL;DR

This paper introduces a robust differentially private clustered federated learning algorithm that accurately identifies client clusters despite the noise introduced by differential privacy, improving handling of structured data heterogeneity.

Contribution

The paper proposes a novel DPFL algorithm that clusters clients using model updates and loss values, employing GMM and large batch sizes to mitigate DP noise effects.

Findings

Effective in high heterogeneity scenarios

Robust to DP noise with theoretical guarantees

Improves clustering accuracy in privacy-sensitive settings

Abstract

Federated learning (FL), which is a decentralized machine learning (ML) approach, often incorporates differential privacy (DP) to provide rigorous data privacy guarantees. Previous works attempted to address high structured data heterogeneity in vanilla FL settings through clustering clients (a.k.a clustered FL), but these methods remain sensitive and prone to errors, further exacerbated by the DP noise. This vulnerability makes the previous methods inappropriate for differentially private FL (DPFL) settings with structured data heterogeneity. To address this gap, we propose an algorithm for differentially private clustered FL, which is robust to the DP noise in the system and identifies the underlying clients' clusters correctly. To this end, we propose to cluster clients based on both their model updates and training loss values. Furthermore, for clustering clients' model updates at the end of the first round, our proposed approach addresses the server's uncertainties by employing large batch sizes as well as Gaussian Mixture Models (GMM) to reduce the impact of DP and stochastic noise and avoid potential clustering errors. This idea is efficient especially in privacy-sensitive scenarios with more DP noise. We provide theoretical analysis to justify our approach and evaluate it across diverse data distributions and privacy budgets. Our experimental results show its effectiveness in addressing large structured data heterogeneity in DPFL.