Exploring Trade-offs in Dynamic Task Triggering for Loosely Coupled Scientific Workflows

TL;DR

This paper investigates the overheads of data-driven dynamic task triggering in scientific workflows, analyzing various design choices and providing practical guidance for constructing efficient, flexible workflows.

Contribution

It offers a comprehensive evaluation of overheads in dynamic task triggering and presents insights for optimizing data-driven scientific workflows.

Findings

Overheads vary with data size and distribution.

Certain design choices reduce triggering overhead.

Guidelines for constructing efficient workflows are provided.

Abstract

In order to achieve near-time insights, scientific workflows tend to be organized in a flexible and dynamic way. Data-driven triggering of tasks has been explored as a way to support workflows that evolve based on the data. However, the overhead introduced by such dynamic triggering of tasks is an under-studied topic. This paper discusses different facets of dynamic task triggers. Particularly, we explore different ways of constructing a data-driven dynamic workflow and then evaluate the overheads introduced by such design decisions. We evaluate workflows with varying data size, percentage of interesting data, temporal data distribution, and number of tasks triggered. Finally, we provide advice based upon analysis of the evaluation results for users looking to construct data-driven scientific workflows.