A Bespoke Workflow Management System for Data-Driven Urgent HPC
Gordon P. S. Gibb, Nick Brown, Rupert W, Nash, Miguel Mendes, Santiago, Monedero, Humberto D\'iaz Fidalgo, Joaqu\'in Ram\'irez Cisneros, Adri\'an, Cardil, Max Kontak

TL;DR
This paper introduces a novel workflow management system designed for urgent HPC scenarios, enabling real-time data integration and adaptive decision-making during disaster response, addressing limitations of existing systems.
Contribution
The paper presents a new adaptive workflow management system tailored for urgent computing, integrating real-time data to refine predictions during disasters, developed for the VESTEC project.
Findings
Successfully integrated into VESTEC system
Demonstrated adaptability to real-time data
Improved decision support in urgent scenarios
Abstract
In this paper we present a workflow management system which permits the kinds of data-driven workflows required by urgent computing, namely where new data is integrated into the workflow as a disaster progresses in order refine the predictions as time goes on. This allows the workflow to adapt to new data at runtime, a capability that most workflow management systems do not possess. The workflow management system was developed for the EU-funded VESTEC project, which aims to fuse HPC with real-time data for supporting urgent decision making. We first describe an example workflow from the VESTEC project, and show why existing workflow technologies do not meet the needs of the project. We then go on to present the design of our Workflow Management System, describe how it is implemented into the VESTEC system, and provide an example of the workflow system in use for a test case.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
