Optimal Client Sampling in Federated Learning with Client-Level Heterogeneous Differential Privacy

TL;DR

This paper introduces GDPFed and GDPFed$^+$, novel federated learning algorithms that optimize client sampling and privacy partitioning to enhance model utility under heterogeneous client privacy requirements.

Contribution

The paper proposes GDPFed and GDPFed$^+$, new methods that improve federated learning with client-level differential privacy by grouping clients and optimizing sampling ratios, backed by theoretical analysis.

Findings

GDPFed reduces privacy budget waste and improves utility.

GDPFed$^+$ further enhances performance with model sparsification and optimized sampling.

Empirical results show substantial gains over existing methods.

Abstract

Federated Learning with client-level differential privacy (DP) provides a promising framework for collaboratively training models while rigorously protecting clients' privacy. However, classic approaches like DP-FedAvg struggle when clients have heterogeneous privacy requirements, as they must uniformly enforce the strictest privacy level across all clients, leading to excessive DP noise and significant degradation in model utility. Existing methods to improve the model utility in such heterogeneous privacy settings often assume a trusted server and are largely heuristic, resulting in suboptimal performance and lacking strong theoretical foundations. In this work, we address these challenges under a practical attack model where both clients and the server are honest-but-curious. We propose GDPFed, which partitions clients into groups based on their privacy budgets and achieves client-level DP within each group to reduce the privacy budget waste and hence improve the model utility. Based on the privacy and convergence analysis of GDPFed, we find that the magnitude of DP noise depends on both model dimensionality and the per-group client sampling ratios. To further improve the performance of GDPFed, we introduce GDPFed$^+$, which integrates model sparsification to eliminate unnecessary noise and optimizes per-group client sampling ratios to minimize convergence error. Extensive empirical evaluations on multiple benchmark datasets demonstrate the effectiveness of GDPFed$^+$, showing substantial performance gains compared with state-of-the-art methods.