EMS-FL: Federated Tuning of Mixture-of-Experts in Satellite-Terrestrial Networks via Expert-Driven Model Splitting

TL;DR

EMS-FL introduces an expert-driven federated learning approach tailored for satellite-terrestrial networks, enabling efficient training with intermittent connectivity by assigning relevant experts to device clusters and allowing asynchronous local updates.

Contribution

The paper proposes EMS-FL, a novel federated learning method that uses expert-driven model splitting and asynchronous updates to improve training efficiency and accuracy in satellite-terrestrial networks.

Findings

Faster convergence compared to traditional federated learning.

Higher accuracy achieved with expert-based model splitting.

Reduced training overhead in satellite-terrestrial environments.

Abstract

The rapid advancement of large AI models imposes stringent demands on data volume and computational resources. Federated learning, though designed to exploit distributed data and computational resources, faces data shortage from limited network coverage and computational constraints from edge devices. To address these issues, both the mixture-of-experts (MoE) and satellite-terrestrial network (STN) provide promising solutions, offering lightweight computation overhead and broad coverage, respectively. However, the satellite-ground relative motion results in intermittent connectivity, hindering conventional federated learning that relies on model synchronization across devices. To leverage the coverage of STN while preserving training efficiency, we propose EMS-FL, an expert-driven model splitting and federated learning method. EMS-FL assigns each device cluster only the experts highly correlated to their local data. Through non-overlapping expert assignments, asynchronous local learning is further proposed, where each device cluster trains its assigned experts consecutively and only uploads local parameters to the satellite during connected phases for aggregation and model updates. Consequently, EMS-FL effectively reduces the training overhead and achieves both faster convergence and higher accuracy compared with conventional federated learning. Rigorous convergence analysis is provided to theoretically characterize the learning performance. Furthermore, comprehensive experiments are conducted using public datasets and large models, validating the superiority of EMS-FL.