An Intermediate Data-driven Methodology for Scientific Workflow Management System to Support Reusability

TL;DR

This thesis introduces a novel data-driven methodology for scientific workflow management systems that enhances reusability, reduces execution time, and optimizes data storage through an adaptive, recommendation-based approach tested on real-world workflows.

Contribution

It presents the first adaptive data reuse technique for SWfMS, improving workflow building efficiency and storage cost reduction based on real-world workflow data.

Findings

51% workflow building reusability achieved

74% reduction in workflow execution time

Around 40% reusability with adaptive technique

Abstract

In this thesis first we propose an intermediate data management scheme for a SWfMS. In our second attempt, we explored the possibilities and introduced an automatic recommendation technique for a SWfMS from real-world workflow data (i.e Galaxy [1] workflows) where our investigations show that the proposed technique can facilitate 51% of workflow building in a SWfMS by reusing intermediate data of previous workflows and can reduce 74% execution time of workflow buildings in a SWfMS. Later we propose an adaptive version of our technique by considering the states of tools in a SWfMS, which shows around 40% reusability for workflows. Consequently, in our fourth study, We have done several experiments for analyzing the performance and exploring the effectiveness of the technique in a SWfMS for various environments. The technique is introduced to emphasize on storing cost reduction, increase data reusability, and faster workflow execution, to the best of our knowledge, which is the first of its kind. Detail architecture and evaluation of the technique are presented in this thesis. We believe our findings and developed system will contribute significantly to the research domain of SWfMSs.