Dynamically Provisioning Cray DataWarp Storage

TL;DR

This paper presents a mechanism for dynamically provisioning data management systems like BeeGFS on HPC storage resources, enabling flexible and on-demand storage allocation for complex applications.

Contribution

It introduces a novel method for deploying data management systems dynamically on HPC storage resources, demonstrated with BeeGFS on Cray DataWarp and other systems.

Findings

Successful deployment of BeeGFS across multiple DataWarp nodes.

Mechanism applicable to non-Cray systems.

Enhanced flexibility in storage resource allocation.

Abstract

Complex applications and workflows needs are often exclusively expressed in terms of computational resources on HPC systems. In many cases, other resources like storage or network are not allocatable and are shared across the entire HPC system. By looking at the storage resource in particular, any workflow or application should be able to select both its preferred data manager and its required storage capability or capacity. To achieve such a goal, new mechanisms should be introduced. In this work, we introduce such a mechanism for dynamically provision a data management system on top of storage devices. We particularly focus our effort on deploying a BeeGFS instance across multiple DataWarp nodes on a Cray XC50 system. However, we also demonstrate that the same mechanism can be used to deploy BeeGFS on non-Cray system.