Challenges of Translating HPC codes to Workflows for Heterogeneous and Dynamic Environments

TL;DR

This paper discusses transforming HPC CFD codes into workflows for dynamic, heterogeneous environments using opportunistic scheduling, highlighting challenges and potential for new HPC paradigms like cloud and edge computing.

Contribution

It presents a case study on converting CFD codes into workflows for RedisDG, demonstrating challenges and benefits of dynamic task graph unfolding in heterogeneous environments.

Findings

Dynamic task graph unfolding improves workflow performance.

Heterogeneous platforms enable flexible HPC execution.

Challenges include managing dynamic task scheduling.

Abstract

In this paper we would like to share our experience for transforming a parallel code for a Computational Fluid Dynamics (CFD) problem into a parallel version for the RedisDG workflow engine. This system is able to capture heterogeneous and highly dynamic environments, thanks to opportunistic scheduling strategies. We show how to move to the field of "HPC as a Service" in order to use heterogeneous platforms. We mainly explain, through the CFD use case, how to transform the parallel code and we exhibit challenges to 'unfold' the task graph dynamically in order to improve the overall performance (in a broad sense) of the workflow engine. We discuss in particular of the impact on the workflow engine of such dynamic feature. This paper states that new models for High Performance Computing are possible, under the condition we revisit our mind in the direction of the potential of new paradigms such as cloud, edge computing.