A Survey of Distributed Learning in Cloud, Mobile, and Edge Settings

TL;DR

This survey reviews distributed learning techniques across cloud and edge environments, focusing on data and model parallelism, partitioning schemes, and their trade-offs to optimize resource use and performance.

Contribution

It provides a comprehensive comparison of distributed learning approaches in cloud and edge settings, highlighting partitioning strategies and their impact on efficiency.

Findings

Analyzes various partitioning schemes for different layer types.

Highlights trade-offs between computational efficiency and communication overhead.

Provides insights for future research in distributed learning.

Abstract

In the era of deep learning (DL), convolutional neural networks (CNNs), and large language models (LLMs), machine learning (ML) models are becoming increasingly complex, demanding significant computational resources for both inference and training stages. To address this challenge, distributed learning has emerged as a crucial approach, employing parallelization across various devices and environments. This survey explores the landscape of distributed learning, encompassing cloud and edge settings. We delve into the core concepts of data and model parallelism, examining how models are partitioned across different dimensions and layers to optimize resource utilization and performance. We analyze various partitioning schemes for different layer types, including fully connected, convolutional, and recurrent layers, highlighting the trade-offs between computational efficiency, communication overhead, and memory constraints. This survey provides valuable insights for future research and development in this rapidly evolving field by comparing and contrasting distributed learning approaches across diverse contexts.