Runtime vs Scheduler: Analyzing Dask's Overheads

TL;DR

This paper investigates Dask's overheads, revealing that runtime overheads are the main bottleneck, and introduces a Rust-based server that outperforms Dask despite simpler scheduling.

Contribution

It presents a Rust-based drop-in replacement for Dask's server that reduces runtime overhead and scales better, challenging the focus on scheduler optimization.

Findings

Random scheduler performs competitively with Dask's scheduler.

The main bottleneck in Dask is its runtime overhead.

The Rust implementation scales better and outperforms Dask.

Abstract

Dask is a distributed task framework which is commonly used by data scientists to parallelize Python code on computing clusters with little programming effort. It uses a sophisticated work-stealing scheduler which has been hand-tuned to execute task graphs as efficiently as possible. But is scheduler optimization a worthwhile effort for Dask? Our paper shows on many real world task graphs that even a completely random scheduler is surprisingly competitive with its built-in scheduler and that the main bottleneck of Dask lies in its runtime overhead. We develop a drop-in replacement for the Dask central server written in Rust which is backwards compatible with existing Dask programs. Thanks to its efficient runtime, our server implementation is able to scale up to larger clusters than Dask and consistently outperforms it on a variety of task graphs, despite the fact that it uses a simpler scheduling algorithm.