Practical and Private (Deep) Learning without Sampling or Shuffling

TL;DR

This paper introduces DP-FTRL, a differentially private learning method that avoids the need for sampling or shuffling, making it more practical for federated learning while maintaining strong privacy and accuracy guarantees.

Contribution

The paper proposes a novel DP-FTRL algorithm that eliminates the need for privacy amplification, offering better flexibility in data access patterns compared to DP-SGD.

Findings

DP-FTRL outperforms amplified DP-SGD in privacy/accuracy trade-offs.

DP-FTRL is more flexible for federated learning scenarios.

Theoretical and empirical results validate DP-FTRL's effectiveness.

Abstract

We consider training models with differential privacy (DP) using mini-batch gradients. The existing state-of-the-art, Differentially Private Stochastic Gradient Descent (DP-SGD), requires privacy amplification by sampling or shuffling to obtain the best privacy/accuracy/computation trade-offs. Unfortunately, the precise requirements on exact sampling and shuffling can be hard to obtain in important practical scenarios, particularly federated learning (FL). We design and analyze a DP variant of Follow-The-Regularized-Leader (DP-FTRL) that compares favorably (both theoretically and empirically) to amplified DP-SGD, while allowing for much more flexible data access patterns. DP-FTRL does not use any form of privacy amplification. The code is available at https://github.com/google-research/federated/tree/master/dp_ftrl and https://github.com/google-research/DP-FTRL .