ParallelSFL: A Novel Split Federated Learning Framework Tackling Heterogeneity Issues

TL;DR

ParallelSFL is a split federated learning framework that improves training efficiency, reduces communication, and enhances model accuracy in heterogeneous edge computing environments by clustering workers and optimizing their training strategies.

Contribution

The paper introduces ParallelSFL, a novel split federated learning framework with a clustering strategy and adaptive training to address heterogeneity in edge systems.

Findings

Reduces traffic consumption by at least 21%.

Speeds up training by at least 1.36 times.

Improves model accuracy by at least 5%.

Abstract

Mobile devices contribute more than half of the world's web traffic, providing massive and diverse data for powering various federated learning (FL) applications. In order to avoid the communication bottleneck on the parameter server (PS) and accelerate the training of large-scale models on resourceconstraint workers in edge computing (EC) system, we propose a novel split federated learning (SFL) framework, termed ParallelSFL. Concretely, we split an entire model into a bottom submodel and a top submodel, and divide participating workers into multiple clusters, each of which collaboratively performs the SFL training procedure and exchanges entire models with the PS. However, considering the statistical and system heterogeneity in edge systems, it is challenging to arrange suitable workers to specific clusters for efficient model training. To address these challenges, we carefully develop an effective clustering strategy by optimizing a utility function related to training efficiency and model accuracy. Specifically, ParallelSFL partitions workers into different clusters under the heterogeneity restrictions, thereby promoting model accuracy as well as training efficiency. Meanwhile, ParallelSFL assigns diverse and appropriate local updating frequencies for each cluster to further address system heterogeneity. Extensive experiments are conducted on a physical platform with 80 NVIDIA Jetson devices, and the experimental results show that ParallelSFL can reduce the traffic consumption by at least 21%, speed up the model training by at least 1.36x, and improve model accuracy by at least 5% in heterogeneous scenarios, compared to the baselines.