CycleSL: Server-Client Cyclical Update Driven Scalable Split Learning

TL;DR

CycleSL is a novel split learning framework that improves scalability and performance by using cyclical updates and feature resampling, effectively addressing limitations of existing methods in distributed model training.

Contribution

CycleSL introduces an aggregation-free, cyclical update mechanism inspired by coordinate descent, enhancing scalability and performance in split learning with heterogeneity.

Findings

CycleSL improves model accuracy on non-iid datasets.

CycleSL reduces server resource overhead compared to parallel split learning.

CycleSL demonstrates better convergence and robustness across multiple benchmarks.

Abstract

Split learning emerges as a promising paradigm for collaborative distributed model training, akin to federated learning, by partitioning neural networks between clients and a server without raw data exchange. However, sequential split learning suffers from poor scalability, while parallel variants like parallel split learning and split federated learning often incur high server resource overhead due to model duplication and aggregation, and generally exhibit reduced model performance and convergence owing to factors like client drift and lag. To address these limitations, we introduce CycleSL, a novel aggregation-free split learning framework that enhances scalability and performance and can be seamlessly integrated with existing methods. Inspired by alternating block coordinate descent, CycleSL treats server-side training as an independent higher-level machine learning task, resampling client-extracted features (smashed data) to mitigate heterogeneity and drift. It then performs cyclical updates, namely optimizing the server model first, followed by client updates using the updated server for gradient computation. We integrate CycleSL into previous algorithms and benchmark them on five publicly available datasets with non-iid data distribution and partial client attendance. Our empirical findings highlight the effectiveness of CycleSL in enhancing model performance. Our source code is available at https://gitlab.lrz.de/hctl/CycleSL.