Sizey: Memory-Efficient Execution of Scientific Workflow Tasks

TL;DR

Sizey is an online memory prediction system for scientific workflows that dynamically learns and selects the best model during execution, significantly reducing memory waste and improving resource utilization.

Contribution

It introduces a novel online machine learning approach for memory prediction in workflows, with a new RAQ score for model evaluation and continuous online retraining.

Findings

Median memory waste reduction of 24.68% with Sizey.

Effective online model selection improves resource efficiency.

Validated on six real-world scientific workflows.

Abstract

As the amount of available data continues to grow in fields as diverse as bioinformatics, physics, and remote sensing, the importance of scientific workflows in the design and implementation of reproducible data analysis pipelines increases. When developing workflows, resource requirements must be defined for each type of task in the workflow. Typically, task types vary widely in their computational demands because they are simply wrappers for arbitrary black-box analysis tools. Furthermore, the resource consumption for the same task type can vary considerably as well due to different inputs. Since underestimating memory resources leads to bottlenecks and task failures, workflow developers tend to overestimate memory resources. However, overprovisioning of memory wastes resources and limits cluster throughput. Addressing this problem, we propose Sizey, a novel online memory prediction method for workflow tasks. During workflow execution, Sizey simultaneously trains multiple machine learning models and then dynamically selects the best model for each workflow task. To evaluate the quality of the model, we introduce a novel resource allocation quality (RAQ) score based on memory prediction accuracy and efficiency. Sizey's prediction models are retrained and re-evaluated online during workflow execution, continuously incorporating metrics from completed tasks. Our evaluation with a prototype implementation of Sizey uses metrics from six real-world scientific workflows from the popular nf-core framework and shows a median reduction in memory waste over time of 24.68% compared to the respective best-performing state-of-the-art baseline.