Communication and Computation Reduction for Split Learning using Asynchronous Training

TL;DR

This paper introduces an asynchronous split learning method that reduces communication and computation costs by updating less frequently and quantizing data, maintaining high accuracy and privacy.

Contribution

It proposes a novel asynchronous training scheme with quantization for split learning, significantly lowering communication and computation overhead while preserving privacy.

Findings

Communication cost reduced by up to 106.7x

Client-side computation reduced by up to 32.1x

Accuracy loss less than 0.5% across models and client counts

Abstract

Split learning is a promising privacy-preserving distributed learning scheme that has low computation requirement at the edge device but has the disadvantage of high communication overhead between edge device and server. To reduce the communication overhead, this paper proposes a loss-based asynchronous training scheme that updates the client-side model less frequently and only sends/receives activations/gradients in selected epochs. To further reduce the communication overhead, the activations/gradients are quantized using 8-bit floating point prior to transmission. An added benefit of the proposed communication reduction method is that the computations at the client side are reduced due to reduction in the number of client model updates. Furthermore, the privacy of the proposed communication reduction based split learning method is almost the same as traditional split learning. Simulation results on VGG11, VGG13 and ResNet18 models on CIFAR-10 show that the communication cost is reduced by 1.64x-106.7x and the computations in the client are reduced by 2.86x-32.1x when the accuracy degradation is less than 0.5% for the single-client case. For 5 and 10-client cases, the communication cost reduction is 11.9x and 11.3x on VGG11 for 0.5% loss in accuracy.