Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters

TL;DR

This paper introduces Structured Partial Backpropagation (SPB), a method to reduce resource consumption in distributed deep learning training, and JigSaw, a scheduler that enhances cluster efficiency by up to 28%.

Contribution

The paper presents SPB, a novel backpropagation control technique, and JigSaw, a scheduler that optimizes resource use and improves large-scale cluster efficiency.

Findings

SPB reduces network bandwidth, compute, and memory usage while maintaining model quality.

JigSaw improves cluster efficiency by up to 28%.

The combined approach enhances resource utilization in distributed deep learning.

Abstract

Many organizations employ compute clusters equipped with accelerators such as GPUs and TPUs for training deep learning models in a distributed fashion. Training is resource-intensive, consuming significant compute, memory, and network resources. Many prior works explore how to reduce training resource footprint without impacting quality, but their focus on a subset of the bottlenecks (typically only the network) limits their ability to improve overall cluster utilization. In this work, we exploit the unique characteristics of deep learning workloads to propose Structured Partial Backpropagation(SPB), a technique that systematically controls the amount of backpropagation at individual workers in distributed training. This simultaneously reduces network bandwidth, compute utilization, and memory footprint while preserving model quality. To efficiently leverage the benefits of SPB at cluster level, we introduce JigSaw, a SPB aware scheduler, which does scheduling at the iteration level for Deep Learning Training(DLT) jobs. We find that JigSaw can improve large scale cluster efficiency by as high as 28\%.