FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion

TL;DR

This paper introduces FedGEMS, a federated learning framework where a powerful server model learns from multiple clients and transfers knowledge back, enhancing performance, robustness, and efficiency in resource-constrained environments.

Contribution

It proposes a novel selective knowledge fusion paradigm enabling a server model to improve federated learning by learning from multiple clients and transferring knowledge back to them.

Findings

Achieves superior performance on server and client models.

Provides robustness against poisoning attacks.

Enhances communication efficiency in federated learning.

Abstract

Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints. Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes. In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL. By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance. Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server on various image classification tasks.