Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Survey

TL;DR

This survey reviews recent strategies and challenges in resource allocation and workload scheduling for large-scale distributed deep learning, highlighting key insights and practical case studies to guide future research.

Contribution

It provides a comprehensive overview of recent advances in resource management and scheduling strategies specifically for large-scale distributed deep learning environments.

Findings

Identifies key challenges like scheduling complexity and heterogeneity.

Highlights effective strategies for resource allocation and workload scheduling.

Includes a case study on training large language models.

Abstract

With rapidly increasing distributed deep learning workloads in large-scale data centers, efficient distributed deep learning framework strategies for resource allocation and workload scheduling have become the key to high-performance deep learning. The large-scale environment with large volumes of datasets, models, and computational and communication resources raises various unique challenges for resource allocation and workload scheduling in distributed deep learning, such as scheduling complexity, resource and workload heterogeneity, and fault tolerance. To uncover these challenges and corresponding solutions, this survey reviews the literature, mainly from 2019 to 2024, on efficient resource allocation and workload scheduling strategies for large-scale distributed DL. We explore these strategies by focusing on various resource types, scheduling granularity levels, and performance goals during distributed training and inference processes. We highlight critical challenges for each topic and discuss key insights of existing technologies. To illustrate practical large-scale resource allocation and workload scheduling in real distributed deep learning scenarios, we use a case study of training large language models. This survey aims to encourage computer science, artificial intelligence, and communications researchers to understand recent advances and explore future research directions for efficient framework strategies for large-scale distributed deep learning.