Detailed comparison of communication efficiency of split learning and federated learning

TL;DR

This paper compares the communication efficiency of split learning and federated learning across various scenarios, highlighting when each method is more effective based on client number, model size, and data samples.

Contribution

It provides a comprehensive analysis of communication efficiencies of split learning and federated learning under diverse practical settings and model sizes.

Findings

Split learning outperforms federated learning with more clients or larger models.

Federated learning is more communication-efficient with more data samples and fewer clients.

The study covers models from millions to hundreds of billions of parameters.

Abstract

We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and juxtapose the two methods under various real-life scenarios. We consider settings of small and large number of clients as well as small models (1M - 6M parameters), large models (10M - 200M parameters) and very large models (1 Billion-100 Billion parameters). We show that increasing number of clients or increasing model size favors split learning setup over the federated while increasing the number of data samples while keeping the number of clients or model size low makes federated learning more communication efficient.