Distilling A Universal Expert from Clustered Federated Learning

TL;DR

This paper proposes a federated learning framework that distills a universal expert model from clustered client models, effectively capturing shared knowledge across non-IID data distributions and improving overall performance.

Contribution

It introduces a novel three-step federated learning process that distills a universal expert from cluster-specific models, enhancing shared knowledge transfer and model heterogeneity handling.

Findings

Universal expert improves generalization across clients.

Distillation-based aggregation reduces model conflicts.

Experimental results show superior performance over traditional methods.

Abstract

Clustered Federated Learning (CFL) addresses the challenges posed by non-IID data by training multiple group- or cluster-specific expert models. However, existing methods often overlook the shared information across clusters, which represents the generalizable knowledge valuable to all participants in the Federated Learning (FL) system. To overcome this limitation, this paper introduces a novel FL framework that distills a universal expert model from the knowledge of multiple clusters. This universal expert captures globally shared information across all clients and is subsequently distributed to each client as the initialization for the next round of model training. The proposed FL framework operates in three iterative steps: (1) local model training at each client, (2) cluster-specific model aggregation, and (3) universal expert distillation. This three-step learning paradigm ensures the preservation of fine-grained non-IID characteristics while effectively incorporating shared knowledge across clusters. Compared to traditional gradient-based aggregation methods, the distillation-based model aggregation introduces greater flexibility in handling model heterogeneity and reduces conflicts among cluster-specific experts. Extensive experimental results demonstrate the superior performance of the proposed method across various scenarios, highlighting its potential to advance the state of CFL by balancing personalized and shared knowledge more effectively.