MP-SL: Multihop Parallel Split Learning

TL;DR

MP-SL introduces a multihop parallel split learning framework that enables resource-constrained devices to participate in distributed ML training efficiently by reducing memory requirements and improving system heterogeneity handling.

Contribution

The paper proposes MP-SL, a novel multihop parallel split learning approach that splits models across multiple nodes to reduce memory use and enhance scalability in heterogeneous environments.

Findings

Multihop MP-SL reduces memory demands at compute nodes.

Multihop MP-SL outperforms one-hop setups in heterogeneous systems.

The framework effectively involves resource-constrained devices in collaborative training.

Abstract

Federated Learning (FL) stands out as a widely adopted protocol facilitating the training of Machine Learning (ML) models while maintaining decentralized data. However, challenges arise when dealing with a heterogeneous set of participating devices, causing delays in the training process, particularly among devices with limited resources. Moreover, the task of training ML models with a vast number of parameters demands computing and memory resources beyond the capabilities of small devices, such as mobile and Internet of Things (IoT) devices. To address these issues, techniques like Parallel Split Learning (SL) have been introduced, allowing multiple resource-constrained devices to actively participate in collaborative training processes with assistance from resourceful compute nodes. Nonetheless, a drawback of Parallel SL is the substantial memory allocation required at the compute nodes, for instance training VGG-19 with 100 participants needs 80 GB. In this paper, we introduce Multihop Parallel SL (MP-SL), a modular and extensible ML as a Service (MLaaS) framework designed to facilitate the involvement of resource-constrained devices in collaborative and distributed ML model training. Notably, to alleviate memory demands per compute node, MP-SL supports multihop Parallel SL-based training. This involves splitting the model into multiple parts and utilizing multiple compute nodes in a pipelined manner. Extensive experimentation validates MP-SL's capability to handle system heterogeneity, demonstrating that the multihop configuration proves more efficient than horizontally scaled one-hop Parallel SL setups, especially in scenarios involving more cost-effective compute nodes.