Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generator

TL;DR

FedMD-CG is a privacy-preserving federated learning method that uses knowledge distillation and a conditional generator to improve performance, privacy, and robustness against data heterogeneity.

Contribution

It introduces FedMD-CG, a novel federated learning approach that decouples local models and employs a conditional generator for server aggregation, enhancing privacy and performance.

Findings

FedMD-CG achieves high accuracy on image classification tasks.

The method is robust to data heterogeneity.

It does not require training extra discriminators.

Abstract

Federated Learning (FL) is gaining popularity as a distributed learning framework that only shares model parameters or gradient updates and keeps private data locally. However, FL is at risk of privacy leakage caused by privacy inference attacks. And most existing privacy-preserving mechanisms in FL conflict with achieving high performance and efficiency. Therefore, we propose FedMD-CG, a novel FL method with highly competitive performance and high-level privacy preservation, which decouples each client's local model into a feature extractor and a classifier, and utilizes a conditional generator instead of the feature extractor to perform server-side model aggregation. To ensure the consistency of local generators and classifiers, FedMD-CG leverages knowledge distillation to train local models and generators at both the latent feature level and the logit level. Also, we construct additional classification losses and design new diversity losses to enhance client-side training. FedMD-CG is robust to data heterogeneity and does not require training extra discriminators (like cGAN). We conduct extensive experiments on various image classification tasks to validate the superiority of FedMD-CG.