Federated Learning with Privacy-Preserving Ensemble Attention Distillation

TL;DR

This paper introduces a privacy-preserving federated learning framework that uses ensemble attention distillation with unlabeled public data, achieving competitive performance while reducing privacy risks in medical and image tasks.

Contribution

It proposes a novel FL method utilizing offline knowledge distillation with ensemble attention, enhancing privacy preservation and reducing communication rounds.

Findings

Achieves competitive accuracy on image classification, segmentation, and reconstruction tasks.

Significantly reduces privacy leakage compared to existing FL methods.

Demonstrates robustness with decentralized and heterogeneous data.

Abstract

Federated Learning (FL) is a machine learning paradigm where many local nodes collaboratively train a central model while keeping the training data decentralized. This is particularly relevant for clinical applications since patient data are usually not allowed to be transferred out of medical facilities, leading to the need for FL. Existing FL methods typically share model parameters or employ co-distillation to address the issue of unbalanced data distribution. However, they also require numerous rounds of synchronized communication and, more importantly, suffer from a privacy leakage risk. We propose a privacy-preserving FL framework leveraging unlabeled public data for one-way offline knowledge distillation in this work. The central model is learned from local knowledge via ensemble attention distillation. Our technique uses decentralized and heterogeneous local data like existing FL approaches, but more importantly, it significantly reduces the risk of privacy leakage. We demonstrate that our method achieves very competitive performance with more robust privacy preservation based on extensive experiments on image classification, segmentation, and reconstruction tasks.