Effectively Heterogeneous Federated Learning: A Pairing and Split Learning Based Approach

TL;DR

This paper introduces a novel split federated learning framework that pairs clients based on resources and splits the model to reduce training latency, effectively addressing client heterogeneity in federated learning.

Contribution

It proposes a client pairing strategy and a split learning approach to improve training speed and performance in heterogeneous federated learning environments.

Findings

Significantly improves FL training speed

Effective in IID and Non-IID data distributions

Uses a heuristic greedy algorithm for client pairing

Abstract

As a promising paradigm federated Learning (FL) is widely used in privacy-preserving machine learning, which allows distributed devices to collaboratively train a model while avoiding data transmission among clients. Despite its immense potential, the FL suffers from bottlenecks in training speed due to client heterogeneity, leading to escalated training latency and straggling server aggregation. To deal with this challenge, a novel split federated learning (SFL) framework that pairs clients with different computational resources is proposed, where clients are paired based on computing resources and communication rates among clients, meanwhile the neural network model is split into two parts at the logical level, and each client only computes the part assigned to it by using the SL to achieve forward inference and backward training. Moreover, to effectively deal with the client pairing problem, a heuristic greedy algorithm is proposed by reconstructing the optimization of training latency as a graph edge selection problem. Simulation results show the proposed method can significantly improve the FL training speed and achieve high performance both in independent identical distribution (IID) and Non-IID data distribution.