An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning

TL;DR

This paper introduces FedKTL, a novel upload-efficient knowledge transfer scheme in heterogeneous federated learning that leverages a pre-trained generator to improve knowledge sharing among diverse models with minimal communication overhead.

Contribution

It proposes a new knowledge transfer method using a pre-trained generator to facilitate task-specific knowledge sharing in heterogeneous federated learning.

Findings

FedKTL outperforms seven state-of-the-art methods by up to 7.31%

The scheme is effective across multiple datasets and model architectures

Applicable in cloud-edge scenarios with only one edge client

Abstract

Heterogeneous Federated Learning (HtFL) enables task-specific knowledge sharing among clients with different model architectures while preserving privacy. Despite recent research progress, transferring knowledge in HtFL is still difficult due to data and model heterogeneity. To tackle this, we introduce a public pre-trained generator (e.g., StyleGAN or Stable Diffusion) as the bridge and propose a new upload-efficient knowledge transfer scheme called Federated Knowledge-Transfer-Loop (FedKTL). It can produce task-related prototypical image-vector pairs via the generator's inference on the server. With these pairs, each client can transfer common knowledge from the generator to its local model through an additional supervised local task. We conduct extensive experiments on four datasets under two types of data heterogeneity with 14 heterogeneous models, including CNNs and ViTs. Results show that our FedKTL surpasses seven state-of-the-art methods by up to 7.31%. Moreover, our knowledge transfer scheme is applicable in cloud-edge scenarios with only one edge client. Code: https://github.com/TsingZ0/FedKTL