Testing Research Software: A Survey

TL;DR

This survey examines current testing practices in research software, highlighting challenges faced by developers and suggesting that training and cultural change are essential for improving software trustworthiness.

Contribution

It provides an empirical analysis of research software testing practices and offers recommendations to enhance testing culture and processes.

Findings

Developers report moderate knowledge of testing but face many challenges.

Training and cultural shifts can significantly improve testing practices.

Testing remains difficult due to scientific complexity and community norms.

Abstract

Background: Research software plays an important role in solving real-life problems, empowering scientific innovations, and handling emergency situations. Therefore, the correctness and trustworthiness of research software are of absolute importance. Software testing is an important activity for identifying problematic code and helping to produce high-quality software. However, testing of research software is difficult due to the complexity of the underlying science, relatively unknown results from scientific algorithms, and the culture of the research software community. Aims: The goal of this paper is to better understand current testing practices, identify challenges, and provide recommendations on how to improve the testing process for research software development. Method: We surveyed members of the research software developer community to collect information regarding their knowledge about and use of software testing in their projects. Results: We analysed 120 responses and identified that even though research software developers report they have an average level of knowledge about software testing, they still find it difficult due to the numerous challenges involved. However, there are a number of ways, such as proper training, that can improve the testing process for research software. Conclusions: Testing can be challenging for any type of software. This difficulty is especially present in the development of research software, where software engineering activities are typically given less attention. To produce trustworthy results from research software, there is a need for a culture change so that testing is valued and teams devote appropriate effort to writing and executing tests.