Workflow environments for advanced cyberinfrastructure platforms

TL;DR

This paper advocates for integrated, holistic workflow environments that unify data and computing processes across diverse, distributed cyberinfrastructure platforms, aiming to enhance scientific research efficiency.

Contribution

It proposes a novel, holistic approach to scientific workflows that combines data and compute processes with high-level interfaces and dynamic runtime support for heterogeneous infrastructures.

Findings

Conceptual framework for integrated workflow environments.

Ongoing development steps for holistic workflow tools.

Emphasis on performance and energy efficiency in heterogeneous settings.

Abstract

Progress in science is deeply bound to the effective use of high-performance computing infrastructures and to the efficient extraction of knowledge from vast amounts of data. Such data comes from different sources that follow a cycle composed of pre-processing steps for data curation and preparation for subsequent computing steps, and later analysis and analytics steps applied to the results. However, scientific workflows are currently fragmented in multiple components, with different processes for computing and data management, and with gaps in the viewpoints of the user profiles involved. Our vision is that future workflow environments and tools for the development of scientific workflows should follow a holistic approach, where both data and computing are integrated in a single flow built on simple, high-level interfaces. The topics of research that we propose involve novel ways to express the workflows that integrate the different data and compute processes, dynamic runtimes to support the execution of the workflows in complex and heterogeneous computing infrastructures in an efficient way, both in terms of performance and energy. These infrastructures include highly distributed resources, from sensors and instruments, and devices in the edge, to High-Performance Computing and Cloud computing resources. This paper presents our vision to develop these workflow environments and also the steps we are currently following to achieve it.