Analysis of Workflow Schedulers in Simulated Distributed Environments

TL;DR

This paper introduces an open-source simulation environment for benchmarking workflow schedulers in distributed systems, revealing significant impacts of realistic network modeling and implementation details on scheduler performance.

Contribution

It provides a flexible simulation platform for testing scheduling algorithms and highlights the importance of realistic network models and detailed algorithm descriptions.

Findings

Network models significantly affect scheduling results.

Implementation details greatly influence scheduler performance.

Realistic simulation environments are crucial for accurate benchmarking.

Abstract

Task graphs provide a simple way to describe scientific workflows (sets of tasks with dependencies) that can be executed on both HPC clusters and in the cloud. An important aspect of executing such graphs is the used scheduling algorithm. Many scheduling heuristics have been proposed in existing works; nevertheless, they are often tested in oversimplified environments. We provide an extensible simulation environment designed for prototyping and benchmarking task schedulers, which contains implementations of various scheduling algorithms and is open-sourced, in order to be fully reproducible. We use this environment to perform a comprehensive analysis of workflow scheduling algorithms with a focus on quantifying the effect of scheduling challenges that have so far been mostly neglected, such as delays between scheduler invocations or partially unknown task durations. Our results indicate that network models used by many previous works might produce results that are off by an order of magnitude in comparison to a more realistic model. Additionally, we show that certain implementation details of scheduling algorithms which are often neglected can have a large effect on the scheduler's performance, and they should thus be described in great detail to enable proper evaluation.