GADGET: Online Resource Optimization for Scheduling Ring-All-Reduce Learning Jobs

TL;DR

This paper introduces GADGET, a novel resource scheduling algorithm for ring-all-reduce deep learning jobs, backed by a new analytical model and extensive experiments showing its effectiveness and advantages over existing methods.

Contribution

The paper presents a new analytical model for scheduling ring-all-reduce deep learning jobs and introduces GADGET, a provably effective resource scheduling algorithm.

Findings

GADGET outperforms existing scheduling methods in experiments.

The analytical model effectively captures diverse DDL performance objectives.

GADGET provides strong theoretical performance guarantees.

Abstract

Fueled by advances in distributed deep learning (DDL), recent years have witnessed a rapidly growing demand for resource-intensive distributed/parallel computing to process DDL computing jobs. To resolve network communication bottleneck and load balancing issues in distributed computing, the so-called ``ring-all-reduce'' decentralized architecture has been increasingly adopted to remove the need for dedicated parameter servers. To date, however, there remains a lack of theoretical understanding on how to design resource optimization algorithms for efficiently scheduling ring-all-reduce DDL jobs in computing clusters. This motivates us to fill this gap by proposing a series of new resource scheduling designs for ring-all-reduce DDL jobs. Our contributions in this paper are three-fold: i) We propose a new resource scheduling analytical model for ring-all-reduce deep learning, which covers a wide range of objectives in DDL performance optimization (e.g., excessive training avoidance, energy efficiency, fairness); ii) Based on the proposed performance analytical model, we develop an efficient resource scheduling algorithm called GADGET (greedy ring-all-reduce distributed graph embedding technique), which enjoys a provable strong performance guarantee; iii) We conduct extensive trace-driven experiments to demonstrate the effectiveness of the GADGET approach and its superiority over the state of the art.