Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search

TL;DR

Tessel is an automated system that efficiently searches for optimized distributed execution schedules of large DNN models, significantly improving training speed and inference latency by exploiting repetitive scheduling patterns.

Contribution

Tessel introduces a novel two-phase schedule search method that leverages pattern repetition to optimize distributed DNN execution across diverse operator placement strategies.

Findings

Achieves up to 5.5x training speedup.

Reduces inference latency by up to 38%.

Effectively handles diverse operator placement strategies.

Abstract

Increasingly complex and diverse deep neural network (DNN) models necessitate distributing the execution across multiple devices for training and inference tasks, and also require carefully planned schedules for performance. However, existing practices often rely on predefined schedules that may not fully exploit the benefits of emerging diverse model-aware operator placement strategies. Handcrafting high-efficiency schedules can be challenging due to the large and varying schedule space. This paper presents Tessel, an automated system that searches for efficient schedules for distributed DNN training and inference for diverse operator placement strategies. To reduce search costs, Tessel leverages the insight that the most efficient schedules often exhibit repetitive pattern (repetend) across different data inputs. This leads to a two-phase approach: repetend construction and schedule completion. By exploring schedules for various operator placement strategies, Tessel significantly improves both training and inference performance. Experiments with representative DNN models demonstrate that Tessel achieves up to 5.5x training performance speedup and up to 38% inference latency reduction.