Gain-loss ratio of storing intermediate data from workflows

TL;DR

This paper proposes a method to improve workflow management systems by automatically suggesting and deciding which intermediate data to store, using association rule mining to analyze previous workflows and optimize data handling.

Contribution

It introduces a novel approach that applies association rule mining to recommend and justify data storage decisions in workflow systems, enhancing automation and efficiency.

Findings

Association rule mining effectively suggests subsequent modules for data retrieval and storage.

Gain-loss ratios provide a clear explanation for data storage recommendations.

The approach improves workflow efficiency by optimizing intermediate data management.

Abstract

Sequentially, the systematic processing of a significant amount of data can be necessary for input datasets to get desired outputs. In a workflow management system(WMS), usually, users build workflows by manually selecting and interconnecting various modules concerning some particular tasks. Thus, a system of automatically suggesting the appropriate intermediate datasets for modules and a suggestion for the decision of saving intermediate states will be helpful in a pipeline building process. This work investigates a technique for both automatically suggesting the intermediate datasets to use and store through mining and analyzing association rules from the previously developed workflows. Investigation on workflows shows that the association rule mining technique can help us to suggest subsequent modules for retrieving and storing data and explain them with gain-loss ratios.