Elastic deep learning in multi-tenant GPU cluster

TL;DR

This paper introduces EDL, a low-overhead elastic deep learning framework that dynamically adjusts GPU parallelism to improve cluster efficiency and job performance.

Contribution

The paper presents EDL, a novel approach with techniques like stop-free scaling and dynamic data pipelines to enable efficient elasticity in multi-tenant GPU clusters.

Findings

EDL reduces overhead of parallelism adjustments

EDL improves GPU cluster utilization

EDL benefits scheduling and resource management

Abstract

We study how to support elasticity, i.e., the ability to dynamically adjust the parallelism (number of GPUs), for deep neural network (DNN) training. Elasticity can benefit multi-tenant GPU cluster management in many ways, e.g., achieving various scheduling objectives (e.g., job throughput, job completion time, GPU efficiency) according to cluster load variations, maximizing the use of transient idle resources, performance profiling, job migration, and straggler mitigation. However, existing parallelism adjustment strategies incur high overheads, which hinder many applications from making effective use of elasticity. We propose EDL to enable low-overhead elastic deep learning with a simple API. We present techniques that are necessary to reduce the overhead of parallelism adjustments, such as stop-free scaling and dynamic data pipeline. We also demonstrate that EDL can indeed bring significant benefits to the above-listed applications in GPU cluster management.