Automatic Metadata Capture and Processing for High-Performance Workflows
Polina Shpilker, Line Pouchard
TL;DR
This paper presents a software system for automatically capturing and organizing metadata from high-performance computing workflows to enhance reproducibility and ease of analysis.
Contribution
It introduces a method for automatic metadata collection and compares two formats for storing metadata to improve usability for researchers.
Findings
Metadata collection is feasible on HPC systems.
Two storage formats are evaluated for usability.
Organized metadata facilitates workflow performance analysis.
Abstract
Modern workflows run on increasingly heterogeneous computing architectures and with this heterogeneity comes additional complexity. We aim to apply the FAIR principles for research reproducibility by developing software to collect metadata annotations for workflows run on HPC systems. We experiment with two possible formats to uniformly store these metadata, and reorganize the collected metadata to be as easy to use as possible for researchers studying their workflow performance.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsScientific Computing and Data Management · Semantic Web and Ontologies · Advanced Database Systems and Queries