Data Version Management and Machine-Actionable Reproducibility for HPC

TL;DR

This paper introduces an extension to DataLad that enables data version control and machine-actionable reproducibility in HPC environments using SLURM, addressing compatibility issues with batch processing and improving efficiency.

Contribution

The paper presents a novel extension to DataLad that makes it compatible with HPC batch processing systems like SLURM, enabling concurrent job scheduling on shared data repositories.

Findings

Enables multiple HPC jobs to access data repositories concurrently.

Improves efficiency on parallel file systems in HPC environments.

Ensures reproducibility of data processing workflows in HPC settings.

Abstract

We present a solution for research data version control and machine-actionable reproducibility of data processing for High Performance Computing (HPC) environments and the SLURM batch scheduler. Both aspects are important for research data management and the DataLad tool provides both based on the very prevalent git version control system. However, it is incompatible with HPC batch processing. The presented extension makes it compatible with HPC batch processing with the SLURM scheduler. It solves the fundamental incompatibility so that multiple jobs can be scheduled concurrently on the same data repository. It also avoids inefficient behavior patterns which may emerge on parallel file systems.