KS+: Predicting Workflow Task Memory Usage Over Time

TL;DR

KS+ is a predictive method that estimates task memory usage over time in workflow systems, reducing memory wastage by 38% through dynamic segmentation and input-based predictions.

Contribution

This paper introduces KS+, a novel approach for dynamic memory prediction in workflows, improving resource allocation accuracy over static estimates.

Findings

Achieves 38% reduction in memory wastage.

Outperforms state-of-the-art baseline methods.

Effective on real-world nf-core workflows.

Abstract

Scientific workflow management systems enable the reproducible execution of data analysis pipelines on cluster infrastructures managed by resource managers such as Kubernetes, Slurm, or HTCondor. These resource managers require resource estimates for each workflow task to be executed on one of the cluster nodes. However, task resource consumption varies significantly between different tasks and for the same task with different inputs. Furthermore, resource consumption also fluctuates during a task's execution. As a result, manually configuring static memory allocations is error-prone, often leading users to overestimate memory usage to avoid costly failures from under-provisioning, which results in significant memory wastage. We propose KS+, a method that predicts a task's memory consumption over time depending on its inputs. For this, KS+ dynamically segments the task execution and predicts the memory required for each segment. Our experimental evaluation shows an average reduction in memory wastage of 38% compared to the best-performing state-of-the-art baseline for two real-world workflows from the popular nf-core repository.