Streaming supercomputing needs workflow-enabled programming-in-the-large

TL;DR

Future online supercomputing workloads require integrated, workflow-enabled programming approaches to effectively combine streams, caches, analysis, and simulations for rapid scientific results.

Contribution

This position paper advocates for workflow-enabled programming-in-the-large to manage complex, integrated supercomputing applications involving multiple advanced technologies.

Findings

Highlights the need for coupling streams, caches, analysis, and simulations.

Argues for programming in the large to manage complexity.

Emphasizes the importance of workflow-enabled approaches in supercomputing.

Abstract

This is a position paper, submitted to the Future Online Analysis Platform Workshop (https://press3.mcs.anl.gov/futureplatform/), which argues that simple data analysis applications are common today, but future online supercomputing workloads will need to couple multiple advanced technologies (streams, caches, analysis, and simulations) to rapidly deliver scientific results. Each of these technologies are active research areas when integrated with high-performance computing. These components will interact in complex ways, therefore coupling them needs to be programmed. Programming in the large, on top of existing applications, enables us to build much more capable applications and to productively manage this complexity.