Parallel Split Learning with Global Sampling

TL;DR

This paper introduces GPSL, a scalable parallel split learning method that maintains fixed global batch sizes and improves training stability and accuracy in non-IID data settings, outperforming traditional methods.

Contribution

GPSL is a novel server-driven scheme that fixes global batch size, uses pooled-level proportions for local batch scheduling, and eliminates bias, enhancing scalability and performance in non-IID environments.

Findings

GPSL stabilizes optimization under non-IID splits.

GPSL achieves centralized-like accuracy on CIFAR datasets.

GPSL shortens training time by avoiding data depletion effects.

Abstract

Parallel split learning (PSL) suffers from two intertwined issues: the effective batch size grows with the number of clients, and data that is not identically and independently distributed (non-IID) skews global batches. We present parallel split learning with global sampling (GPSL), a server-driven scheme that fixes the global batch size while computing per-client batch-size schedules using pooled-level proportions. The actual samples are drawn locally without replacement by each selected client. This eliminates per-class rounding, decouples the effective batch from the client count, and makes each global batch distributionally equivalent to centralized uniform sampling without replacement. Consequently, we obtain finite-population deviation guarantees via Serfling's inequality, yielding a zero rounding bias compared to local sampling schemes. GPSL is a drop-in replacement for PSL with negligible overhead and scales to large client populations. In extensive experiments on CIFAR-10/100 and ResNet-18/34 under non-IID splits, GPSL stabilizes optimization and achieves centralized-like accuracy, while fixed local batching trails by up to 60%. Furthermore, GPSL shortens training time by avoiding inflation of training steps induced by data-depletion. These findings suggest GPSL is a promising and scalable approach for learning in resource-constrained environments.