GAPSL: A Gradient-Aligned Parallel Split Learning on Heterogeneous Data

TL;DR

GAPSL introduces a gradient-aligned parallel split learning framework that improves convergence and performance in federated learning with heterogeneous data by aligning client gradients through leader selection and regularization.

Contribution

It proposes a novel gradient alignment method for parallel split learning, addressing divergence issues caused by gradient inconsistency across clients.

Findings

GAPSL outperforms existing methods in training accuracy.

GAPSL reduces training latency.

GAPSL effectively mitigates gradient inconsistency issues.

Abstract

The increasing complexity of neural networks poses significant challenges for democratizing FL on resource?constrained client devices. Parallel split learning (PSL) has emerged as a promising solution by offloading substantial computing workload to a server via model partitioning, shrinking client-side computing load, and eliminating the client-side model aggregation for reduced communication and deployment costs. Since PSL is aggregation-free, it suffers from severe training divergence stemming from gradient directional inconsistency across clients. To address this challenge, we propose GAPSL, a gradient-aligned PSL framework that comprises two key components: leader gradient identification (LGI) and gradient direction alignment (GDA). LGI dynamically selects a set of directionally consistent client gradients to construct a leader gradient that captures the global convergence trend. GDA employs a direction-aware regularization to align each client's gradient with the leader gradient, thereby mitigating inter-device gradient directional inconsistency and enhancing model convergence. We evaluate GAPSL on a prototype computing testbed. Extensive experiments demonstrate that GAPSL consistently outperforms state-of-the-art benchmarks in training accuracy and latency.