FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation

TL;DR

FedMoE-DA introduces a federated learning framework using Mixture of Experts with domain-aware aggregation and peer-to-peer synchronization, improving personalization, robustness, and reducing communication costs.

Contribution

This work presents a novel federated learning method combining MoE architecture with domain-aware fine-grained aggregation and P2P communication to enhance efficiency and personalization.

Findings

Reduces communication overhead significantly.

Achieves high model performance and personalization.

Enhances robustness against data heterogeneity.

Abstract

Federated learning (FL) is a collaborative machine learning approach that enables multiple clients to train models without sharing their private data. With the rise of deep learning, large-scale models have garnered significant attention due to their exceptional performance. However, a key challenge in FL is the limitation imposed by clients with constrained computational and communication resources, which hampers the deployment of these large models. The Mixture of Experts (MoE) architecture addresses this challenge with its sparse activation property, which reduces computational workload and communication demands during inference and updates. Additionally, MoE facilitates better personalization by allowing each expert to specialize in different subsets of the data distribution. To alleviate the communication burdens between the server and clients, we propose FedMoE-DA, a new FL model training framework that leverages the MoE architecture and incorporates a novel domain-aware, fine-grained aggregation strategy to enhance the robustness, personalizability, and communication efficiency simultaneously. Specifically, the correlation between both intra-client expert models and inter-client data heterogeneity is exploited. Moreover, we utilize peer-to-peer (P2P) communication between clients for selective expert model synchronization, thus significantly reducing the server-client transmissions. Experiments demonstrate that our FedMoE-DA achieves excellent performance while reducing the communication pressure on the server.