DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models

TL;DR

This paper introduces DP-FedAdamW, an optimizer designed for differentially private federated learning, addressing variance, bias, and client drift issues to improve convergence and privacy guarantees.

Contribution

It proposes the first AdamW-based optimizer for DP federated learning, with theoretical convergence guarantees and empirical improvements on vision and language models.

Findings

Outperforms SOTA by 5.83% on Tiny-ImageNet with ε=1.

Establishes an unbiased second-moment estimator under DP.

Proves linearly accelerated convergence without heterogeneity assumptions.

Abstract

Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter $(\varepsilon,\delta)$-DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, $\varepsilon=1$), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83\%. The code is available in Appendix.