Communication and Storage Efficient Federated Split Learning

TL;DR

This paper introduces CSE-FSL, a federated split learning approach that significantly reduces communication and storage costs by using an auxiliary network and selective data transmission, while maintaining high accuracy.

Contribution

The paper proposes a novel CSE-FSL strategy that minimizes communication and server storage requirements, with theoretical convergence guarantees and improved efficiency over existing methods.

Findings

Significant reduction in communication overhead compared to existing FSL methods.

Maintains state-of-the-art convergence and accuracy in real-world FL tasks.

Theoretical proof of convergence for non-convex loss functions.

Abstract

Federated learning (FL) is a popular distributed machine learning (ML) paradigm, but is often limited by significant communication costs and edge device computation capabilities. Federated Split Learning (FSL) preserves the parallel model training principle of FL, with a reduced device computation requirement thanks to splitting the ML model between the server and clients. However, FSL still incurs very high communication overhead due to transmitting the smashed data and gradients between the clients and the server in each global round. Furthermore, the server has to maintain separate models for every client, resulting in a significant computation and storage requirement that grows linearly with the number of clients. This paper tries to solve these two issues by proposing a communication and storage efficient federated and split learning (CSE-FSL) strategy, which utilizes an auxiliary network to locally update the client models while keeping only a single model at the server, hence avoiding the communication of gradients from the server and greatly reducing the server resource requirement. Communication cost is further reduced by only sending the smashed data in selected epochs from the clients. We provide a rigorous theoretical analysis of CSE-FSL that guarantees its convergence for non-convex loss functions. Extensive experimental results demonstrate that CSE-FSL has a significant communication reduction over existing FSL techniques while achieving state-of-the-art convergence and model accuracy, using several real-world FL tasks.