Differentially Private Learning with Adaptive Clipping

TL;DR

This paper introduces an adaptive clipping method for differentially private federated learning that estimates the clipping norm online at a specified quantile, improving privacy-utility trade-offs without hyperparameter tuning.

Contribution

The authors propose a novel adaptive clipping technique that dynamically estimates the clipping norm during training, enhancing privacy guarantees and model performance in federated learning.

Findings

Adaptive clipping outperforms fixed clipping in various tasks.

The method requires minimal additional privacy budget.

It is compatible with other federated learning techniques.

Abstract

Existing approaches for training neural networks with user-level differential privacy (e.g., DP Federated Averaging) in federated learning (FL) settings involve bounding the contribution of each user's model update by clipping it to some constant value. However there is no good a priori setting of the clipping norm across tasks and learning settings: the update norm distribution depends on the model architecture and loss, the amount of data on each device, the client learning rate, and possibly various other parameters. We propose a method wherein instead of a fixed clipping norm, one clips to a value at a specified quantile of the update norm distribution, where the value at the quantile is itself estimated online, with differential privacy. The method tracks the quantile closely, uses a negligible amount of privacy budget, is compatible with other federated learning technologies such as compression and secure aggregation, and has a straightforward joint DP analysis with DP-FedAvg. Experiments demonstrate that adaptive clipping to the median update norm works well across a range of realistic federated learning tasks, sometimes outperforming even the best fixed clip chosen in hindsight, and without the need to tune any clipping hyperparameter.