Technical Debt in the Peer-Review Documentation of R Packages: a rOpenSci Case Study

TL;DR

This study investigates technical debt in the peer-review documentation of R packages, revealing prevalent documentation debt and role-specific differences, thus pioneering analysis in scientific software peer-review.

Contribution

It introduces a taxonomy of technical debt in R package peer-reviews and provides the first empirical analysis of this kind in scientific software.

Findings

Documentation debt is the most common type of technical debt.

Different roles report different types of technical debt.

The study provides a publicly available dataset for future research.

Abstract

Context: Technical Debt is a metaphor used to describe code that is "not quite right." Although TD studies have gained momentum, TD has yet to be studied as thoroughly in non-Object-Oriented (OO) or scientific software such as R. R is a multi-paradigm programming language, whose popularity in data science and statistical applications has amplified in recent years. Due to R's inherent ability to expand through user-contributed packages, several community-led organizations were created to organize and peer-review packages in a concerted effort to increase their quality. Nonetheless, it is well-known that most R users do not have a technical programming background, being from multiple disciplines. Objective: The goal of this study is to investigate TD in the peer-review documentation of R packages led by rOpenSci. Method: We collected over 5000 comments from 157 packages that had been reviewed and approved to be published at rOpenSci. We manually analyzed a sample dataset of these comments posted by package authors, editors of rOpenSci, and reviewers during the review process to investigate the TD types present in these reviews. Results: The findings of our study include (i) a taxonomy of TD derived from our analysis of the peer-reviews (ii) documentation debt as being the most prevalent type of debt (iii) different user roles are concerned with different types of TD. For instance, reviewers tend to report some TD types more than other roles, and the TD types they report are different from those reported by the authors of a package. Conclusion: TD analysis in scientific software or peer-review is almost non-existent. Our study is a pioneer but within the context of R packages. However, our findings can serve as a starting point for replication studies, given our public datasets, to perform similar analyses in other scientific software or to investigate the rationale behind our findings.