Reshi: Recommending Resources for Scientific Workflow Tasks on Heterogeneous Infrastructures

TL;DR

Reshi is a resource recommendation method for scientific workflows that improves scheduling efficiency on heterogeneous infrastructures by using regression models to predict optimal task-node assignments.

Contribution

Reshi introduces a regression-based approach for recommending task-node assignments, handling heterogeneity in both tasks and infrastructure, and demonstrates improved scheduling performance.

Findings

Reshi reduces mean makespan by up to 18% compared to HEFT.

Reshi effectively handles heterogeneous resources and tasks.

Benchmarking on AWS shows practical applicability.

Abstract

Scientific workflows typically comprise a multitude of different processing steps which often are executed in parallel on different partitions of the input data. These executions, in turn, must be scheduled on the compute nodes of the computational infrastructure at hand. This assignment is complicated by the facts that (a) tasks typically have highly heterogeneous resource requirements and (b) in many infrastructures, compute nodes offer highly heterogeneous resources. In consequence, predictions of the runtime of a given task on a given node, as required by many scheduling algorithms, are often rather imprecise, which can lead to sub-optimal scheduling decisions. We propose Reshi, a method for recommending task-node assignments during workflow execution that can cope with heterogeneous tasks and heterogeneous nodes. Reshi approaches the problem as a regression task, where task-node pairs are modeled as feature vectors over the results of dedicated micro benchmarks and past task executions. Based on these features, Reshi trains a regression tree model to rank and recommend nodes for each ready-to-run task, which can be used as input to a scheduler. For our evaluation, we benchmarked 27 AWS machine types using three representative workflows. We compare Reshi's recommendations with three state-of-the-art schedulers. Our evaluation shows that Reshi outperforms HEFT by a mean makespan reduction of 7.18% and 18.01% assuming a mean task runtime prediction error of 15%.