Staged deployment of interactive multi-application HPC workflows
Wouter Klijn, Sandra Diaz-Pier, Abigail Morrison, Alexander Peyser

TL;DR
This paper presents a middleware system that simplifies deploying and managing complex, multi-application scientific workflows on HPC systems, enabling advanced neuroscience research and machine learning applications.
Contribution
It introduces a staged, user-centric deployment model that integrates and extends existing workflow management solutions for HPC, tailored to neuroscience and machine learning use cases.
Findings
Supports coupled neuronal simulators with visualization
Enables closed-loop workflows with machine learning and robotics
Reduces deployment complexity for large-scale HPC workflows
Abstract
Running scientific workflows on a supercomputer can be a daunting task for a scientific domain specialist. Workflow management solutions (WMS) are a standard method for reducing the complexity of application deployment on high performance computing (HPC) infrastructure. We introduce the design for a middleware system that extends and combines the functionality from existing solutions in order to create a high-level, staged user-centric operation/deployment model. This design addresses the requirements of several use cases in the life sciences, with a focus on neuroscience. In this manuscript we focus on two use cases: 1) three coupled neuronal simulators (for three different space/time scales) with in-transit visualization and 2) a closed-loop workflow optimized by machine learning, coupling a robot with a neural network simulation. We provide a detailed overview of the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
