FedAUXfdp: Differentially Private One-Shot Federated Distillation

TL;DR

FedAUXfdp introduces a fully differentially private federated distillation method that effectively handles non-iid data, maintaining high accuracy with strong privacy guarantees in just one communication round.

Contribution

This work presents FedAUXfdp, a novel fully differentially private federated distillation approach with theoretical sensitivity bounds and superior performance on large-scale image datasets.

Findings

FedAUXfdp outperforms state-of-the-art baselines under strong privacy constraints.

Full privatization causes negligible accuracy loss across data heterogeneity levels.

The method achieves high accuracy in a single communication round.

Abstract

Federated learning suffers in the case of non-iid local datasets, i.e., when the distributions of the clients' data are heterogeneous. One promising approach to this challenge is the recently proposed method FedAUX, an augmentation of federated distillation with robust results on even highly heterogeneous client data. FedAUX is a partially $(\epsilon, \delta)$-differentially private method, insofar as the clients' private data is protected in only part of the training it takes part in. This work contributes a fully differentially private modification, termed FedAUXfdp. We further contribute an upper bound on the $l_2$-sensitivity of regularized multinomial logistic regression. In experiments with deep networks on large-scale image datasets, FedAUXfdp with strong differential privacy guarantees performs significantly better than other equally privatized SOTA baselines on non-iid client data in just a single communication round. Full privatization of the modified method results in a negligible reduction in accuracy at all levels of data heterogeneity.