The Impact of Cut Layer Selection in Split Federated Learning

TL;DR

This paper analyzes how the choice of cut layer impacts the performance and convergence of split federated learning, revealing that SFL-V2 is sensitive to this choice while SFL-V1 is not, and demonstrating SFL-V2's advantages over FedAvg.

Contribution

It provides the first comprehensive quantitative and theoretical analysis of cut layer effects in split federated learning, highlighting differences between SFL variants.

Findings

SFL-V1 performance is invariant to cut layer choice.

Cut layer selection significantly impacts SFL-V2 performance.

SFL-V2 with proper cut layer outperforms FedAvg on heterogeneous data.

Abstract

Split Federated Learning (SFL) is a distributed machine learning paradigm that combines federated learning and split learning. In SFL, a neural network is partitioned at a cut layer, with the initial layers deployed on clients and remaining layers on a training server. There are two main variants of SFL: SFL-V1 where the training server maintains separate server-side models for each client, and SFL-V2 where the training server maintains a single shared model for all clients. While existing studies have focused on algorithm development for SFL, a comprehensive quantitative analysis of how the cut layer selection affects model performance remains unexplored. This paper addresses this gap by providing numerical and theoretical analysis of SFL performance and convergence relative to cut layer selection. We find that SFL-V1 is relatively invariant to the choice of cut layer, which is consistent with our theoretical results. Numerical experiments on four datasets and two neural networks show that the cut layer selection significantly affects the performance of SFL-V2. Moreover, SFL-V2 with an appropriate cut layer selection outperforms FedAvg on heterogeneous data.