Quantifying the Performance of Federated Transfer Learning

TL;DR

This paper evaluates the efficiency of Federated Transfer Learning (FTL) in real-world settings, identifying key bottlenecks like communication, encryption, and network conditions that impact performance.

Contribution

It provides a quantitative analysis of FTL deployment on Google Cloud, highlighting critical infrastructure factors affecting efficiency and scalability.

Findings

Inter-process communication is the main bottleneck.

Data encryption significantly increases computation overhead.

Network conditions greatly influence large model performance.

Abstract

The scarcity of data and isolated data islands encourage different organizations to share data with each other to train machine learning models. However, there are increasing concerns on the problems of data privacy and security, which urges people to seek a solution like Federated Transfer Learning (FTL) to share training data without violating data privacy. FTL leverages transfer learning techniques to utilize data from different sources for training, while achieving data privacy protection without significant accuracy loss. However, the benefits come with a cost of extra computation and communication consumption, resulting in efficiency problems. In order to efficiently deploy and scale up FTL solutions in practice, we need a deep understanding on how the infrastructure affects the efficiency of FTL. Our paper tries to answer this question by quantitatively measuring a real-world FTL implementation FATE on Google Cloud. According to the results of carefully designed experiments, we verified that the following bottlenecks can be further optimized: 1) Inter-process communication is the major bottleneck; 2) Data encryption adds considerable computation overhead; 3) The Internet networking condition affects the performance a lot when the model is large.