Framework and Methodology for Verification of a Complex Scientific Simulation Software, Flash-X

TL;DR

This paper presents a comprehensive verification methodology for Flash-X, a complex scientific simulation software, addressing challenges in testing composable components to ensure scientific accuracy and software quality.

Contribution

It introduces a systematic approach for verifying highly modular scientific software, balancing quality, productivity, and scientific objectives.

Findings

Developed a methodology for test suite construction in composable software

Achieved improved code coverage for Flash-X components

Provided guidelines for balancing verification with scientific flexibility

Abstract

Computational science relies on scientific software as its primary instrument for scientific discovery. Therefore, similar to the use of other types of scientific instruments, correct software and the correct operation of the software is necessary for executing rigorous scientific investigations. Scientific software verification can be especially difficult, as users typically need to modify the software as part of a scientific study. Systematic methodologies for building test suites for scientific software are rare in the literature. Here, we describe a methodology that we have developed for Flash-X, a community simulation software for multiple scientific domains, that has composable components that can be permuted and combined in a multitude of ways to generate a wide range of applications. Ensuring sufficient code coverage by a test suite is particularly challenging due to this composability. Our methodology includes a consideration of trade-offs between meeting software quality goals, developer productivity, and meeting the scientific goals of the Flash-X user community.