Testing Research Software: An In-Depth Survey of Practices, Methods, and Tools

TL;DR

This survey examines current testing practices in research software, highlighting challenges, tool usage gaps, and the need for specialized testing tools to improve software correctness and development efficiency.

Contribution

It provides an in-depth analysis of testing practices, challenges, and tool needs in research software, emphasizing the gap between existing industrial tools and research-specific requirements.

Findings

Research software testing practices are highly variable.

Main challenges include test case design and output correctness evaluation.

Developers lack familiarity with existing testing tools and need specialized solutions.

Abstract

Context: Research software is essential for developing advanced tools and models to solve complex research problems and drive innovation across domains. Therefore, it is essential to ensure its correctness. Software testing plays a vital role in this task. However, testing research software is challenging due to the software's complexity and to the unique culture of the research software community. Aims: Building on previous research, this study provides an in-depth investigation of testing practices in research software, focusing on test case design, challenges with expected outputs, use of quality metrics, execution methods, tools, and desired tool features. Additionally, we explore whether demographic factors influence testing processes. Method: We survey research software developers to understand how they design test cases, handle output challenges, use metrics, execute tests, and select tools. Results: Research software testing varies widely. The primary challenges are test case design, evaluating test quality, and evaluating the correctness of test outputs. Overall, research software developers are not familiar with existing testing tools and have a need for new tools to support their specific needs. Conclusion: Allocating human resources to testing and providing developers with knowledge about effective testing techniques are important steps toward improving the testing process of research software. While many industrial testing tools exist, they are inadequate for testing research software due to its complexity, specialized algorithms, continuous updates, and need for flexible, custom testing approaches. Access to a standard set of testing tools that address these special characteristics will increase level of testing in research software development and reduce the overhead of distributing knowledge about software testing.