NSC-SL: A Bandwidth-Aware Neural Subspace Compression for Communication-Efficient Split Learning

TL;DR

NSC-SL introduces a bandwidth-aware adaptive compression method for split learning, dynamically optimizing low-rank approximations and tensor factorization to significantly reduce communication overhead while maintaining model accuracy.

Contribution

It presents a novel adaptive compression algorithm that dynamically adjusts to bandwidth constraints and minimizes information loss in split learning.

Findings

Achieves high compression ratios with minimal accuracy loss.

Outperforms existing communication-efficient split learning methods.

Demonstrates effectiveness across various neural network architectures.

Abstract

The expanding scale of neural networks poses a major challenge for distributed machine learning, particularly under limited communication resources. While split learning (SL) alleviates client computational burden by distributing model layers between clients and server, it incurs substantial communication overhead from frequent transmission of intermediate activations and gradients. To tackle this issue, we propose NSC-SL, a bandwidth-aware adaptive compression algorithm for communication-efficient SL. NSC-SL first dynamically determines the optimal rank of low-rank approximation based on the singular value distribution for adapting real-time bandwidth constraints. Then, NSC-SL performs error-compensated tensor factorization using alternating orthogonal iteration with residual feedback, effectively minimizing truncation loss. The collaborative mechanisms enable NSC-SL to achieve high compression ratios while preserving semantic-rich information essential for convergence. Extensive experiments demonstrate the superb performance of NSC-SL.