EspressoDB: A scientific database for managing high-performance computing workflow

TL;DR

EspressoDB is a Python-based framework designed to streamline and manage complex scientific computing workflows, ensuring data integrity and reducing manual effort at high-performance computing facilities.

Contribution

It introduces a novel object-relational data management system tailored for scientific workflows, enhancing flexibility and ease of use compared to existing solutions.

Findings

Improves workflow management efficiency in scientific computing.

Centralizes data storage and guarantees data integrity.

Reduces human time spent on managing computational jobs.

Abstract

Leadership computing facilities around the world support cutting-edge scientific research across a broad spectrum of disciplines including understanding climate change, combating opioid addiction, or simulating the decay of a neutron. While the increase in computational power has allowed scientists to better evaluate the underlying model, the size of these computational projects have grown to a point where a framework is desired to facilitate managing the workflow. A typical scientific computing workflow includes: Defining all input parameters for every step of the computation; Defining dependencies of computational tasks; Storing some of the output data; Post-processing these data files; Performing data analysis on output. EspressoDB is a programmatic object-relational data management framework implemented in Python and based on the Django web framework. EspressoDB was developed to streamline data management workflows, centralize and guarantee data integrity, while providing domain flexibility and ease of use. The framework provided by EspressoDB aims to support the ever increasing complexity of workflows of scientific computing at leadership computing facilities, with the goal of reducing the amount of human time required to manage the jobs, thus giving scientists more time to focus on science.