Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing

TL;DR

This paper proposes a GPU job scheduling method that enables multiple deep learning jobs to share GPUs efficiently, reducing job completion times and improving resource utilization without preemption.

Contribution

It introduces a novel scheduling model and heuristic algorithm, SJF-BSBF, that optimally shares GPU resources among jobs while maintaining training accuracy.

Findings

Reduces average job completion time by 27-33% compared to state-of-the-art schedulers.

Outperforms aggressive GPU sharing baseline by up to 17% in large-scale traces.

Effectively balances resource sharing benefits with deep learning convergence requirements.

Abstract

Deep learning (DL) has demonstrated significant success across diverse fields, leading to the construction of dedicated GPU accelerators within GPU clusters for high-quality training services. Efficient scheduler designs for such clusters are vital to reduce operational costs and enhance resource utilization. While recent schedulers have shown impressive performance in optimizing DL job performance and cluster utilization through periodic reallocation or selection of GPU resources, they also encounter challenges such as preemption and migration overhead, along with potential DL accuracy degradation. Nonetheless, few explore the potential benefits of GPU sharing to improve resource utilization and reduce job queuing times. Motivated by these insights, we present a job scheduling model allowing multiple jobs to share the same set of GPUs without altering job training settings. We introduce SJF-BSBF (shortest job first with best sharing benefit first), a straightforward yet effective heuristic scheduling algorithm. SJF-BSBF intelligently selects job pairs for GPU resource sharing and runtime settings (sub-batch size and scheduling time point) to optimize overall performance while ensuring DL convergence accuracy through gradient accumulation. In experiments with both physical DL workloads and trace-driven simulations, even as a preemption-free policy, SJF-BSBF reduces the average job completion time by 27-33\% relative to the state-of-the-art preemptive DL schedulers. Moreover, SJF-BSBF can wisely determine the optimal resource sharing settings, such as the sharing time point and sub-batch size for gradient accumulation, outperforming the aggressive GPU sharing approach (baseline SJF-FFS policy) by up to 17\% in large-scale traces.