No Peek: A Survey of private distributed deep learning

TL;DR

This survey reviews privacy-preserving distributed deep learning techniques, comparing federated, split, and large batch methods alongside cryptographic and differential privacy approaches, highlighting their benefits, limitations, and future directions.

Contribution

It provides a comprehensive comparison of existing private distributed deep learning methods, analyzing their trade-offs and potential future trends.

Findings

Federated learning balances privacy and model performance.

Cryptographic methods offer strong privacy but incur high computational costs.

Trade-offs exist between privacy, efficiency, and communication in distributed learning.

Abstract

We survey distributed deep learning models for training or inference without accessing raw data from clients. These methods aim to protect confidential patterns in data while still allowing servers to train models. The distributed deep learning methods of federated learning, split learning and large batch stochastic gradient descent are compared in addition to private and secure approaches of differential privacy, homomorphic encryption, oblivious transfer and garbled circuits in the context of neural networks. We study their benefits, limitations and trade-offs with regards to computational resources, data leakage and communication efficiency and also share our anticipated future trends.