Task Runtime Prediction in Scientific Workflows Using an Online Incremental Learning Approach

TL;DR

This paper introduces an online incremental learning method for predicting task runtimes in scientific workflows on cloud platforms, utilizing real-time resource monitoring data to enhance accuracy.

Contribution

It presents a novel online learning approach that adaptively predicts task runtimes using time-series resource data, outperforming existing regression-based methods.

Findings

Improves prediction accuracy by up to 29.89% over state-of-the-art methods.

Utilizes fine-grained resource monitoring data for better modeling.

Demonstrates effectiveness in near real-time cloud environments.

Abstract

Many algorithms in workflow scheduling and resource provisioning rely on the performance estimation of tasks to produce a scheduling plan. A profiler that is capable of modeling the execution of tasks and predicting their runtime accurately, therefore, becomes an essential part of any Workflow Management System (WMS). With the emergence of multi-tenant Workflow as a Service (WaaS) platforms that use clouds for deploying scientific workflows, task runtime prediction becomes more challenging because it requires the processing of a significant amount of data in a near real-time scenario while dealing with the performance variability of cloud resources. Hence, relying on methods such as profiling tasks' execution data using basic statistical description (e.g., mean, standard deviation) or batch offline regression techniques to estimate the runtime may not be suitable for such environments. In this paper, we propose an online incremental learning approach to predict the runtime of tasks in scientific workflows in clouds. To improve the performance of the predictions, we harness fine-grained resources monitoring data in the form of time-series records of CPU utilization, memory usage, and I/O activities that are reflecting the unique characteristics of a task's execution. We compare our solution to a state-of-the-art approach that exploits the resources monitoring data based on regression machine learning technique. From our experiments, the proposed strategy improves the performance, in terms of the error, up to 29.89%, compared to the state-of-the-art solutions.