Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy

TL;DR

This paper introduces Clip21-SGD2M, a novel federated learning algorithm combining clipping, momentum, and error feedback, achieving both optimal differential privacy and strong optimization guarantees in heterogeneous, non-convex settings.

Contribution

The paper proposes Clip21-SGD2M, the first method to simultaneously attain optimal differential privacy and convergence guarantees in federated learning with heterogeneous data.

Findings

Outperforms baselines in non-convex logistic regression

Achieves near-optimal differential privacy guarantees

Demonstrates superior optimization performance in neural network training

Abstract

Strong Differential Privacy (DP) and Optimization guarantees are two desirable properties for a method in Federated Learning (FL). However, existing algorithms do not achieve both properties at once: they either have optimal DP guarantees but rely on restrictive assumptions such as bounded gradients/bounded data heterogeneity, or they ensure strong optimization performance but lack DP guarantees. To address this gap in the literature, we propose and analyze a new method called Clip21-SGD2M based on a novel combination of clipping, heavy-ball momentum, and Error Feedback. In particular, for non-convex smooth distributed problems with clients having arbitrarily heterogeneous data, we prove that Clip21-SGD2M has optimal convergence rate and also near optimal (local-)DP neighborhood. Our numerical experiments on non-convex logistic regression and training of neural networks highlight the superiority of Clip21-SGD2M over baselines in terms of the optimization performance for a given DP-budget.