Statistical Limits and Efficient Algorithms for Differentially Private Federated Learning

TL;DR

This paper introduces new algorithms for differentially private federated learning, balancing estimation accuracy, privacy, and communication efficiency, with theoretical bounds and empirical validation.

Contribution

It proposes FedHybrid and FedNewton algorithms that improve accuracy and reduce communication costs in DP federated learning, with theoretical analysis and practical experiments.

Findings

FedNewton achieves accuracy comparable to FedSGD with fewer communication rounds.

FedHybrid improves accuracy with reduced communication by combining FedSGD and FedAvg.

Theoretical bounds establish the optimality gap for private federated estimators.

Abstract

Federated Learning is a leading framework for training ML and AI models collaboratively across numerous user devices or databases. We study the trade-offs among estimation accuracy, privacy constraints, and communication cost for differentially private (DP) federated M estimation. The two standard methods in the literature are FedAvg, which may suffer from high federation bias, and FedSGD, which can incur high communication cost. Aimed at improving accuracy at a reduced communication cost, we propose FedHybrid, which uses FedSGD starting with an improved initialization by the FedAvg estimator. We propose FedNewton, which averages local Newton iterations to reduce bias in FedAvg, achieving an estimation accuracy comparable to FedSGD with much fewer communication rounds when the number of clients grows sufficiently slowly. We establish finite sample upper bounds on the mean-squared error rates of the DP versions of these estimators as functions of the number of clients, local sample sizes, privacy budget, and number of iterations. We further derive a minimax lower bound on the MSE of any iterative private federated procedure that provides a benchmark to assess the optimality gap of these methods. We numerically evaluate our methods for training a logistic regression and a neural network on the computer vision datasets MNIST and CIFAR-10.