Characterizing the Usefulness of Code Review Comments in Scientific Software for Software Quality and Scientific Rigor

TL;DR

This study analyzes the usefulness of code review comments in scientific open-source software, revealing that 6-33% are not useful and confirming many characteristics found in general-purpose software reviews.

Contribution

It characterizes CR comment usefulness in scientific software, extending prior research and providing insights specific to Sci-OSS communities.

Findings

6-33% of CR comments in Sci-OSS are not useful.

Negative emoji reactions correlate with less useful comments.

Subjective or negative comments are generally not useful.

Abstract

Context: Innovation thrives on scientific software, with useful code review feedback enhancing its correctness and impact. However, unlike general-purpose commercial and open-source software, the usefulness of code review feedback (CR comment) in scientific software remains largely unstudied. Objective: This paper aims to characterize the usefulness of CR comment in scientific opens ource software (Sci-OSS), leveraging existing research on useful CR comment. Method: To achieve this objective, we mine successful Sci-OSS from GitHub, analyze their CR comments with usefulness related features, and compare the findings from prior research on general-purpose commercial and open-source CR comments. Results: The investigation on the usefulness of CR comments in SciOSS confirms many characteristics that prior research identified in general-purpose software. For example, subjective or negative CR comments remain not useful for the Sci-OSS. We also find CR comments which receive negative emoji reactions have a very small correlation with not useful comments, whereas the positive emojis show mixed correlations. Importantly, 6-33% CR comments in Sci-OSS are not useful in our mined repositories. Conclusions: Our investigation into Sci-OSS extends findings from CR comments' usefulness research on general-purpose software, benefiting developers, scientists, and researchers in the Sci-OSS community.